[2025-11-13 08:04:10,035][mllm.models.large_language_model_local][INFO] - Initializing adapter 'agent_adapter': no initial weights provided or found; starting from scratch.
[2025-11-13 08:04:10,820][mllm.models.adapter_training_wrapper][INFO] - Adapter 'agent_adapter': initialized with fresh weights (no initial weights found).
[2025-11-13 08:04:10,827][mllm.models.large_language_model_local][INFO] - Initializing adapter 'critic_adapter': no initial weights provided or found; starting from scratch.
[2025-11-13 08:04:11,919][mllm.models.adapter_training_wrapper][INFO] - Adapter 'critic_adapter': initialized with fresh weights (no initial weights found).
[2025-11-13 08:06:23,301][__main__][INFO] - Starting iteration 0.
[2025-11-13 08:06:23,305][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:06:23,306][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:06:28,108][__main__][INFO] - Number of regex retries in iteration 0: 0
[2025-11-13 08:06:28,109][__main__][INFO] - agents played in iteration 0 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:06:28,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 37.45%, Block Peak % of device VRAM: 18.68%, ΔTime: 00:00:00
[2025-11-13 08:06:28,669][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:06:28,669][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:06:29,337][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:06:29,983][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:06:30,308][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:06:30,634][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:06:30,961][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:06:31,287][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:06:31,608][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:06:31,934][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:06:32,262][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:06:32,585][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:06:32,907][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:06:33,231][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:06:33,557][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:06:33,883][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:06:34,208][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:06:34,531][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:06:34,856][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:06:35,179][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:06:35,501][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:06:35,830][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:06:36,156][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:06:36,482][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:06:36,804][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:06:37,127][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:06:37,450][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:06:37,776][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:06:38,098][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:06:38,423][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:06:38,747][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:06:39,069][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:06:39,392][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:06:39,717][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:06:40,044][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:06:40,814][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 4.58%, Current % of VRAM taken: 42.03%, Block Peak % of device VRAM: 25.21%, ΔTime: 00:00:11
[2025-11-13 08:06:41,485][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:06:41,487][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:06:41,488][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:06:42,792][__main__][INFO] - Iteration 1 took 19s (24.64% Gen, 68.66% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 11m 30s. Estimated total time: 16h 14m 24s. Time estimates for 10 more iterations: 3m 14s, 100 more iterations: 32m 28s, 500 more iterations: 2h 42m 24s.
[2025-11-13 08:06:42,794][__main__][INFO] - Starting iteration 1.
[2025-11-13 08:06:42,798][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:06:42,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:06:46,530][__main__][INFO] - Number of regex retries in iteration 1: 0
[2025-11-13 08:06:46,531][__main__][INFO] - agents played in iteration 1 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:06:46,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,042][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:06:47,085][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:06:47,085][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:06:47,830][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:06:48,126][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:06:48,452][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:06:48,775][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:06:49,104][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:06:49,428][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:06:49,755][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:06:50,079][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:06:50,406][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:06:50,730][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:06:51,061][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:06:51,388][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:06:51,715][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:06:52,044][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:06:52,375][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:06:52,699][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:06:53,030][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:06:53,358][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:06:53,686][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:06:54,010][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:06:54,337][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:06:54,669][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:06:54,992][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:06:55,315][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:06:55,641][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:06:55,977][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:06:56,301][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:06:56,629][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:06:56,956][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:06:57,291][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:06:57,617][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:06:57,944][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:06:58,269][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:06:59,015][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:06:59,753][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:06:59,755][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:06:59,757][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:00,836][__main__][INFO] - Iteration 2 took 18s (20.69% Gen, 73.32% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 58m 43s. Estimated total time: 15h 1m 55s. Time estimates for 10 more iterations: 3m 0s, 100 more iterations: 30m 3s, 500 more iterations: 2h 30m 19s.
[2025-11-13 08:07:00,838][__main__][INFO] - Starting iteration 2.
[2025-11-13 08:07:00,842][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:00,843][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:04,540][__main__][INFO] - Number of regex retries in iteration 2: 0
[2025-11-13 08:07:04,541][__main__][INFO] - agents played in iteration 2 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:07:05,017][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,100][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:05,139][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:05,140][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:05,874][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:07:06,172][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:07:06,499][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:07:06,825][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:07:07,149][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:07:07,473][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:07:07,800][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:07:08,124][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:07:08,448][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:07:08,773][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:07:09,100][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:07:09,425][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:07:09,750][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:07:10,074][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:07:10,402][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:07:10,726][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:07:11,054][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:07:11,378][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:07:11,705][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:07:12,030][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:07:12,355][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:07:12,680][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:07:13,006][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:07:13,331][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:07:13,657][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:07:13,984][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:07:14,309][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:07:14,633][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:07:14,957][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:07:15,281][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:07:15,609][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:07:15,934][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:07:16,259][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:07:16,994][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:17,753][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:17,755][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:17,756][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:18,835][__main__][INFO] - Iteration 3 took 17s (20.55% Gen, 73.45% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 56m 10s. Estimated total time: 14h 59m 40s. Time estimates for 10 more iterations: 2m 59s, 100 more iterations: 29m 59s, 500 more iterations: 2h 29m 56s.
[2025-11-13 08:07:18,837][__main__][INFO] - Starting iteration 3.
[2025-11-13 08:07:18,840][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:18,840][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:22,600][__main__][INFO] - Number of regex retries in iteration 3: 0
[2025-11-13 08:07:22,601][__main__][INFO] - agents played in iteration 3 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:07:23,046][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:23,169][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:23,169][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:23,919][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:07:24,216][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:07:24,542][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:07:24,871][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:07:25,202][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:07:25,529][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:07:25,857][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:07:26,184][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:07:26,512][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:07:26,837][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:07:27,162][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:07:27,502][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:07:27,826][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:07:28,152][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:07:28,479][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:07:28,808][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:07:29,133][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:07:29,458][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:07:29,783][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:07:30,112][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:07:30,437][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:07:30,762][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:07:31,096][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:07:31,425][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:07:31,750][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:07:32,078][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:07:32,410][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:07:32,739][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:07:33,067][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:07:33,395][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:07:33,724][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:07:34,049][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:07:34,373][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:07:35,106][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:35,857][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:35,859][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:35,861][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:36,934][__main__][INFO] - Iteration 4 took 18s (20.78% Gen, 73.28% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 0m 57s. Estimated total time: 15h 4m 45s. Time estimates for 10 more iterations: 3m 0s, 100 more iterations: 30m 9s, 500 more iterations: 2h 30m 47s.
[2025-11-13 08:07:36,936][__main__][INFO] - Starting iteration 4.
[2025-11-13 08:07:36,939][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:36,940][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:40,675][__main__][INFO] - Number of regex retries in iteration 4: 0
[2025-11-13 08:07:40,676][__main__][INFO] - agents played in iteration 4 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:07:41,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,170][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,212][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:41,253][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:41,253][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:42,011][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:07:42,308][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:07:42,636][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:07:42,964][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:07:43,289][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:07:43,613][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:07:43,937][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:07:44,261][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:07:44,599][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:07:44,927][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:07:45,251][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:07:45,578][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:07:45,911][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:07:46,235][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:07:46,559][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:07:46,886][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:07:47,216][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:07:47,540][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:07:47,864][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:07:48,188][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:07:48,514][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:07:48,837][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:07:49,162][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:07:49,486][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:07:49,813][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:07:50,140][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:07:50,464][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:07:50,794][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:07:51,119][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:07:51,451][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:07:51,776][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:07:52,104][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:07:52,431][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:07:53,160][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:07:53,935][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:07:53,937][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:07:53,939][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:07:54,950][__main__][INFO] - Iteration 5 took 18s (20.74% Gen, 73.63% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 56m 29s. Estimated total time: 15h 0m 35s. Time estimates for 10 more iterations: 3m 0s, 100 more iterations: 30m 1s, 500 more iterations: 2h 30m 5s.
[2025-11-13 08:07:54,952][__main__][INFO] - Starting iteration 5.
[2025-11-13 08:07:54,955][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:07:54,956][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:07:58,644][__main__][INFO] - Number of regex retries in iteration 5: 0
[2025-11-13 08:07:58,645][__main__][INFO] - agents played in iteration 5 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:07:59,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:07:59,223][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:07:59,223][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:07:59,964][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:00,260][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:00,586][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:00,911][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:01,238][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:01,563][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:01,891][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:02,219][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:02,544][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:02,869][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:03,193][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:03,518][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:03,846][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:04,174][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:04,506][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:04,838][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:05,162][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:05,487][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:08:05,812][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:08:06,139][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:08:06,467][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:08:06,791][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:08:07,116][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:08:07,441][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:08:07,767][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:08:08,091][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:08:08,419][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:08:08,744][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:08:09,068][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:08:09,393][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:08:09,718][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:08:10,043][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:08:10,369][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:08:11,091][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:08:11,843][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:11,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:11,846][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:13,098][__main__][INFO] - Iteration 6 took 18s (20.33% Gen, 72.76% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 2m 45s. Estimated total time: 15h 7m 10s. Time estimates for 10 more iterations: 3m 1s, 100 more iterations: 30m 14s, 500 more iterations: 2h 31m 11s.
[2025-11-13 08:08:13,100][__main__][INFO] - Starting iteration 6.
[2025-11-13 08:08:13,103][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:08:13,103][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:16,782][__main__][INFO] - Number of regex retries in iteration 6: 0
[2025-11-13 08:08:16,782][__main__][INFO] - agents played in iteration 6 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:08:17,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:17,369][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:17,369][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:18,088][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:18,385][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:18,712][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:19,036][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:19,363][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:19,690][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:20,022][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:20,346][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:20,677][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:21,005][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:21,333][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:21,662][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:21,988][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:22,316][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:22,640][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:22,967][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:23,295][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:23,623][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:08:23,951][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:08:24,276][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:08:24,601][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:08:24,927][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:08:25,252][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:08:25,579][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:08:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:08:26,229][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:08:26,556][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:08:26,883][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:08:27,216][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:08:27,545][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:08:27,877][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:08:28,202][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:08:28,535][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:08:29,263][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:08:30,018][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:30,020][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:30,022][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:31,142][__main__][INFO] - Iteration 7 took 18s (20.39% Gen, 73.39% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 57m 18s. Estimated total time: 15h 2m 1s. Time estimates for 10 more iterations: 3m 0s, 100 more iterations: 30m 4s, 500 more iterations: 2h 30m 20s.
[2025-11-13 08:08:31,145][__main__][INFO] - Starting iteration 7.
[2025-11-13 08:08:31,148][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:08:31,148][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:34,898][__main__][INFO] - Number of regex retries in iteration 7: 0
[2025-11-13 08:08:34,899][__main__][INFO] - agents played in iteration 7 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:08:35,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:35,472][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:35,473][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:36,220][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:36,516][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:36,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:37,167][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:37,495][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:37,821][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:38,149][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:38,474][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:38,798][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:39,123][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:39,449][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:39,775][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:40,100][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:40,425][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:40,751][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:41,083][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:41,411][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:41,736][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:08:42,064][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:08:42,394][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:08:42,727][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:08:43,055][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:08:43,383][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:08:43,713][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:08:44,034][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:08:44,365][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:08:44,694][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:08:45,023][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:08:45,344][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:08:45,669][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:08:45,995][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:08:46,328][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:08:46,653][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:08:47,382][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:08:48,143][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:08:48,144][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:08:48,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:08:49,318][__main__][INFO] - Iteration 8 took 18s (20.64% Gen, 72.90% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 3m 33s. Estimated total time: 15h 8m 33s. Time estimates for 10 more iterations: 3m 1s, 100 more iterations: 30m 17s, 500 more iterations: 2h 31m 25s.
[2025-11-13 08:08:49,321][__main__][INFO] - Starting iteration 8.
[2025-11-13 08:08:49,324][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:08:49,325][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:08:53,062][__main__][INFO] - Number of regex retries in iteration 8: 0
[2025-11-13 08:08:53,063][__main__][INFO] - agents played in iteration 8 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:08:53,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,631][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:08:53,632][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:08:53,632][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:08:54,358][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:08:54,654][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:08:54,979][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:08:55,304][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:08:55,629][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:08:55,954][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:08:56,278][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:08:56,604][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:08:56,934][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:08:57,260][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:08:57,588][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:08:57,913][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:08:58,240][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:08:58,565][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:08:58,891][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:08:59,215][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:08:59,543][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:08:59,868][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:00,196][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:00,524][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:00,849][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:01,173][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:01,500][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:01,825][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:02,153][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:02,478][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:02,811][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:03,140][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:03,465][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:03,794][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:04,121][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:09:04,446][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:09:04,770][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:09:05,490][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:09:06,267][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:06,268][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:06,270][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:07,275][__main__][INFO] - Iteration 9 took 17s (20.82% Gen, 73.57% Train). Generation: 3s, Training: 13s. Estimated remaining time: 14h 52m 16s. Estimated total time: 14h 57m 35s. Time estimates for 10 more iterations: 2m 59s, 100 more iterations: 29m 55s, 500 more iterations: 2h 29m 35s.
[2025-11-13 08:09:07,277][__main__][INFO] - Starting iteration 9.
[2025-11-13 08:09:07,280][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:09:07,280][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:11,022][__main__][INFO] - Number of regex retries in iteration 9: 0
[2025-11-13 08:09:11,023][__main__][INFO] - agents played in iteration 9 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:09:11,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,567][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:11,608][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:11,608][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:12,350][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:09:12,648][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:09:12,974][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:09:13,299][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:09:13,624][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:09:13,950][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:09:14,276][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:09:14,605][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:09:14,930][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:09:15,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:09:15,594][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:09:15,918][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:09:16,243][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:09:16,567][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:09:16,901][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:09:17,226][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:09:17,551][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:09:17,876][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:18,215][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:18,540][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:18,865][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:19,191][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:19,526][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:19,853][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:20,180][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:20,506][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:20,833][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:21,159][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:21,486][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:21,814][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:22,141][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:09:22,467][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:09:22,793][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:09:23,542][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:09:24,292][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:24,293][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:24,295][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:25,509][__main__][INFO] - Iteration 10 took 18s (20.53% Gen, 72.80% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 5m 53s. Estimated total time: 15h 11m 30s. Time estimates for 10 more iterations: 3m 2s, 100 more iterations: 30m 23s, 500 more iterations: 2h 31m 55s.
[2025-11-13 08:09:25,511][__main__][INFO] - Starting iteration 10.
[2025-11-13 08:09:25,514][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 0 and human policies 1.
[2025-11-13 08:09:25,514][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:29,305][__main__][INFO] - Number of regex retries in iteration 10: 0
[2025-11-13 08:09:29,306][__main__][INFO] - agents played in iteration 10 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:09:29,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:29,901][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:29,901][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:30,668][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:09:30,966][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:09:31,292][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:09:31,633][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:09:31,962][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:09:32,291][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:09:32,616][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:09:32,942][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:09:33,267][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:09:33,593][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:09:33,924][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:09:34,248][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:09:34,573][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:09:34,902][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:09:35,236][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:09:35,555][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:09:35,884][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:09:36,210][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:36,549][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:36,872][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:37,201][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:37,526][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:37,851][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:38,176][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:38,503][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:38,827][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:39,152][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:39,478][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:39,803][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:40,129][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:09:40,454][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:09:40,779][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:09:41,110][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:09:41,852][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:09:42,608][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:09:42,609][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:09:42,611][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:09:44,591][__main__][INFO] - Iteration 11 took 19s (19.87% Gen, 69.74% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 48m 0s. Estimated total time: 15h 53m 55s. Time estimates for 10 more iterations: 3m 10s, 100 more iterations: 31m 47s, 500 more iterations: 2h 38m 59s.
[2025-11-13 08:09:44,594][__main__][INFO] - Starting iteration 11.
[2025-11-13 08:09:44,597][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:09:44,598][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:09:49,055][__main__][INFO] - Number of regex retries in iteration 11: 0
[2025-11-13 08:09:49,055][__main__][INFO] - agents played in iteration 11 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:09:49,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:09:49,630][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:09:49,631][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:09:50,398][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:09:50,696][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:09:51,023][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:09:51,349][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:09:51,674][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:09:52,002][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:09:52,328][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:09:52,656][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:09:52,980][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:09:53,306][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:09:53,631][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:09:53,959][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:09:54,284][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:09:54,611][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:09:54,938][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:09:55,270][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:09:55,599][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:09:55,925][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:09:56,250][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:09:56,575][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:09:56,901][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:09:57,233][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:09:57,561][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:09:57,887][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:09:58,212][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:09:58,541][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:09:58,874][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:09:59,205][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:09:59,531][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:09:59,856][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:00,184][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:00,518][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:00,844][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:01,586][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:02,356][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:02,358][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:02,359][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:03,374][__main__][INFO] - Iteration 12 took 18s (23.74% Gen, 70.85% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 32m 38s. Estimated total time: 15h 38m 52s. Time estimates for 10 more iterations: 3m 7s, 100 more iterations: 31m 17s, 500 more iterations: 2h 36m 28s.
[2025-11-13 08:10:03,376][__main__][INFO] - Starting iteration 12.
[2025-11-13 08:10:03,380][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:03,380][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:07,470][__main__][INFO] - Number of regex retries in iteration 12: 0
[2025-11-13 08:10:07,470][__main__][INFO] - agents played in iteration 12 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:10:07,932][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:07,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:08,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:08,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:08,053][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:08,054][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:08,818][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:10:09,115][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:10:09,446][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:10:09,766][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:10:10,091][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:10:10,423][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:10:10,751][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:10:11,074][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:10:11,399][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:10:11,726][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:10:12,052][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:10:12,380][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:10:12,706][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:10:13,033][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:10:13,358][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:10:13,686][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:10:14,014][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:10:14,346][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:10:14,679][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:10:15,017][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:10:15,345][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:10:15,678][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:10:16,002][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:10:16,337][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:10:16,662][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:10:16,990][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:10:17,316][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:10:17,644][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:10:17,970][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:10:18,294][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:18,620][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:18,946][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:19,274][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:20,022][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:20,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:20,768][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:20,770][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:21,775][__main__][INFO] - Iteration 13 took 18s (22.23% Gen, 72.30% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 13m 13s. Estimated total time: 15h 19m 46s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 39s, 500 more iterations: 2h 33m 17s.
[2025-11-13 08:10:21,777][__main__][INFO] - Starting iteration 13.
[2025-11-13 08:10:21,781][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:21,781][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:25,881][__main__][INFO] - Number of regex retries in iteration 13: 0
[2025-11-13 08:10:25,881][__main__][INFO] - agents played in iteration 13 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:10:26,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:26,453][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:26,454][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:27,198][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:10:27,494][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:10:27,823][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:10:28,148][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:10:28,475][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:10:28,799][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:10:29,128][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:10:29,450][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:10:29,776][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:10:30,102][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:10:30,430][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:10:30,753][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:10:31,079][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:10:31,407][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:10:31,739][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:10:32,064][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:10:32,390][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:10:32,716][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:10:33,041][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:10:33,370][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:10:33,696][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:10:34,022][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:10:34,351][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:10:34,673][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:10:34,998][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:10:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:10:35,652][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:10:35,981][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:10:36,306][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:10:36,632][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:36,958][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:37,284][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:37,612][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:38,351][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:39,121][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:39,122][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:39,124][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:40,148][__main__][INFO] - Iteration 14 took 18s (22.32% Gen, 72.09% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 11m 34s. Estimated total time: 15h 18m 25s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 36s, 500 more iterations: 2h 33m 4s.
[2025-11-13 08:10:40,150][__main__][INFO] - Starting iteration 14.
[2025-11-13 08:10:40,153][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:40,154][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:10:44,291][__main__][INFO] - Number of regex retries in iteration 14: 0
[2025-11-13 08:10:44,292][__main__][INFO] - agents played in iteration 14 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:10:44,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:10:44,873][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:10:44,873][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:10:45,628][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:10:45,925][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:10:46,253][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:10:46,577][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:10:46,906][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:10:47,231][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:10:47,559][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:10:47,888][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:10:48,217][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:10:48,542][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:10:48,868][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:10:49,193][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:10:49,521][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:10:49,853][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:10:50,178][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:10:50,505][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:10:50,832][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:10:51,164][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:10:51,492][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:10:51,818][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:10:52,146][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:10:52,472][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:10:52,798][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:10:53,139][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:10:53,468][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:10:53,801][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:10:54,130][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:10:54,457][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:10:54,785][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:10:55,110][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:10:55,441][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:10:55,772][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:10:56,100][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:10:56,847][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:10:57,604][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:10:57,605][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:10:57,607][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:10:58,693][__main__][INFO] - Iteration 15 took 18s (22.32% Gen, 71.82% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 19m 52s. Estimated total time: 15h 27m 2s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 54s, 500 more iterations: 2h 34m 30s.
[2025-11-13 08:10:58,695][__main__][INFO] - Starting iteration 15.
[2025-11-13 08:10:58,699][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:10:58,699][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:02,820][__main__][INFO] - Number of regex retries in iteration 15: 0
[2025-11-13 08:11:02,821][__main__][INFO] - agents played in iteration 15 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:11:03,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,358][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:03,398][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:03,398][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:04,145][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:04,441][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:11:04,768][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:11:05,094][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:11:05,419][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:11:05,752][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:11:06,081][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:11:06,407][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:11:06,736][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:11:07,064][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:11:07,390][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:11:07,715][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:11:08,040][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:11:08,368][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:11:08,694][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:11:09,022][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:11:09,348][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:11:09,674][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:11:10,001][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:11:10,326][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:11:10,652][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:11:10,980][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:11:11,305][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:11:11,631][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:11:11,956][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:11:12,282][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:11:12,607][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:11:12,933][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:11:13,259][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:11:13,587][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:11:13,913][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:11:14,239][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:11:14,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:11:15,301][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:11:16,213][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:16,215][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:16,217][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:17,397][__main__][INFO] - Iteration 16 took 18s (22.04% Gen, 71.64% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 27m 30s. Estimated total time: 15h 34m 59s. Time estimates for 10 more iterations: 3m 6s, 100 more iterations: 31m 9s, 500 more iterations: 2h 35m 49s.
[2025-11-13 08:11:17,399][__main__][INFO] - Starting iteration 16.
[2025-11-13 08:11:17,402][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:11:17,402][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:21,437][__main__][INFO] - Number of regex retries in iteration 16: 0
[2025-11-13 08:11:21,438][__main__][INFO] - agents played in iteration 16 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:11:21,892][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:21,932][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:21,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:22,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:22,013][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:22,014][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:22,752][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:23,048][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:11:23,374][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:11:23,705][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:11:24,033][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:11:24,362][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:11:24,690][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:11:25,017][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:11:25,349][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:11:25,679][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:11:26,004][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:11:26,333][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:11:26,666][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:11:26,995][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:11:27,322][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:11:27,650][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:11:27,976][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:11:28,308][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:11:28,636][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:11:28,962][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:11:29,288][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:11:29,618][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:11:29,944][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:11:30,270][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:11:30,596][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:11:30,924][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:11:31,249][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:11:31,577][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:11:31,905][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:11:32,230][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:11:32,556][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:11:32,881][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:11:33,208][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:11:33,950][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:11:34,691][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:34,692][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:34,694][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:35,736][__main__][INFO] - Iteration 17 took 18s (22.01% Gen, 72.30% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 8m 58s. Estimated total time: 15h 16m 45s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 33s, 500 more iterations: 2h 32m 47s.
[2025-11-13 08:11:35,738][__main__][INFO] - Starting iteration 17.
[2025-11-13 08:11:35,741][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:11:35,741][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:39,774][__main__][INFO] - Number of regex retries in iteration 17: 0
[2025-11-13 08:11:39,775][__main__][INFO] - agents played in iteration 17 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:11:40,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,356][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:40,356][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:40,356][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:41,094][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:41,392][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:11:41,717][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:11:42,041][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:11:42,368][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:11:42,692][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:11:43,020][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:11:43,345][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:11:43,674][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:11:44,001][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:11:44,328][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:11:44,656][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:11:44,990][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:11:45,318][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:11:45,647][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:11:45,974][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:11:46,299][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:11:46,625][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:11:46,954][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:11:47,282][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:11:47,615][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:11:47,947][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:11:48,275][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:11:48,603][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:11:48,929][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:11:49,257][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:11:49,584][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:11:49,912][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:11:50,239][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:11:50,568][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:11:50,892][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:11:51,225][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:11:51,554][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:11:52,280][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:11:53,029][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:11:53,030][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:11:53,032][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:11:54,183][__main__][INFO] - Iteration 18 took 18s (21.87% Gen, 71.88% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 14m 4s. Estimated total time: 15h 22m 10s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 44s, 500 more iterations: 2h 33m 41s.
[2025-11-13 08:11:54,186][__main__][INFO] - Starting iteration 18.
[2025-11-13 08:11:54,188][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:11:54,189][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:11:58,217][__main__][INFO] - Number of regex retries in iteration 18: 0
[2025-11-13 08:11:58,218][__main__][INFO] - agents played in iteration 18 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:11:58,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:11:58,803][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:11:58,804][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:11:59,553][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:11:59,849][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:00,174][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:00,499][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:00,827][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:01,152][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:01,478][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:01,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:02,133][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:02,459][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:02,788][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:03,113][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:03,440][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:12:03,767][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:12:04,093][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:12:04,419][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:12:04,743][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:12:05,069][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:12:05,394][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:12:05,719][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:12:06,045][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:12:06,376][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:12:06,702][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:12:07,030][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:12:07,360][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:12:07,692][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:12:08,018][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:12:08,343][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:12:08,669][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:12:08,995][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:12:09,321][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:12:09,645][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:12:09,975][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:12:10,722][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:12:11,492][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:11,493][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:11,495][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:12,499][__main__][INFO] - Iteration 19 took 18s (22.00% Gen, 72.51% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 7m 10s. Estimated total time: 15h 15m 34s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 31s, 500 more iterations: 2h 32m 35s.
[2025-11-13 08:12:12,501][__main__][INFO] - Starting iteration 19.
[2025-11-13 08:12:12,504][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:12:12,505][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:16,583][__main__][INFO] - Number of regex retries in iteration 19: 0
[2025-11-13 08:12:16,584][__main__][INFO] - agents played in iteration 19 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:12:17,048][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,089][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,129][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:17,169][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:17,169][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:17,944][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:12:18,240][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:18,571][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:18,904][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:19,232][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:19,558][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:19,885][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:20,211][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:20,542][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:20,875][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:21,204][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:21,535][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:21,868][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:12:22,196][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:12:22,522][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:12:22,850][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:12:23,181][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:12:23,503][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:12:23,832][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:12:24,161][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:12:24,487][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:12:24,816][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:12:25,149][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:12:25,474][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:12:25,803][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:12:26,142][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:12:26,470][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:12:26,796][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:12:27,125][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:12:27,460][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:12:27,788][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:12:28,115][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:12:28,448][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:12:29,203][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:12:29,975][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:29,976][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:29,978][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:31,011][__main__][INFO] - Iteration 20 took 18s (22.04% Gen, 72.37% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 16m 39s. Estimated total time: 15h 25m 22s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 50s, 500 more iterations: 2h 34m 13s.
[2025-11-13 08:12:31,013][__main__][INFO] - Starting iteration 20.
[2025-11-13 08:12:31,017][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 1 and human policies 1.
[2025-11-13 08:12:31,018][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:35,102][__main__][INFO] - Number of regex retries in iteration 20: 0
[2025-11-13 08:12:35,103][__main__][INFO] - agents played in iteration 20 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:12:35,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:35,703][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:35,703][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:36,461][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:12:36,759][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:37,090][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:37,419][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:37,747][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:38,079][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:38,407][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:38,736][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:39,061][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:39,387][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:39,712][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:40,040][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:12:40,373][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:12:40,702][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:12:41,028][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:12:41,355][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:12:41,681][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:12:42,006][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:12:42,331][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:12:42,659][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:12:42,985][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:12:43,312][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:12:43,640][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:12:43,968][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:12:44,293][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:12:44,618][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:12:44,943][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:12:45,272][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:12:45,601][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:12:45,930][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:12:46,257][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:12:46,585][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:12:46,912][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:12:47,649][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:12:48,411][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:12:48,412][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:12:48,414][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:12:50,806][__main__][INFO] - Iteration 21 took 19s (20.64% Gen, 67.26% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 20m 28s. Estimated total time: 16h 29m 30s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 59s, 500 more iterations: 2h 44m 55s.
[2025-11-13 08:12:50,808][__main__][INFO] - Starting iteration 21.
[2025-11-13 08:12:50,811][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:12:50,812][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:12:55,069][__main__][INFO] - Number of regex retries in iteration 21: 0
[2025-11-13 08:12:55,070][__main__][INFO] - agents played in iteration 21 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:12:55,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,582][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:12:55,663][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:12:55,664][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:12:56,414][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:12:56,710][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:12:57,036][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:12:57,364][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:12:57,690][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:12:58,016][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:12:58,341][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:12:58,670][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:12:58,995][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:12:59,320][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:12:59,647][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:12:59,972][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:00,303][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:00,631][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:00,955][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:01,281][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:01,607][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:01,933][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:02,259][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:02,590][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:02,912][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:03,238][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:03,565][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:03,891][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:04,220][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:04,546][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:04,874][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:13:05,200][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:13:05,526][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:13:05,851][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:13:06,177][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:13:06,509][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:13:06,839][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:13:07,587][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:13:08,348][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:08,350][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:08,351][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:09,329][__main__][INFO] - Iteration 22 took 18s (22.99% Gen, 71.72% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 16m 38s. Estimated total time: 15h 25m 58s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 51s, 500 more iterations: 2h 34m 19s.
[2025-11-13 08:13:09,331][__main__][INFO] - Starting iteration 22.
[2025-11-13 08:13:09,334][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:13:09,335][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:13,412][__main__][INFO] - Number of regex retries in iteration 22: 0
[2025-11-13 08:13:13,413][__main__][INFO] - agents played in iteration 22 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:13:13,907][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:13,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:13,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:14,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:14,031][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:14,032][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:14,797][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:13:15,094][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:13:15,421][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:13:15,749][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:13:16,077][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:13:16,403][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:13:16,735][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:13:17,064][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:13:17,394][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:13:17,720][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:13:18,044][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:13:18,370][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:18,697][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:19,022][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:19,347][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:19,678][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:20,001][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:20,326][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:20,652][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:20,977][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:21,305][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:21,634][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:21,959][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:22,291][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:22,612][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:22,940][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:23,264][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:13:23,596][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:13:23,916][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:13:24,241][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:13:24,567][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:13:24,893][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:13:25,220][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:13:25,973][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:13:26,743][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:26,744][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:26,746][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:27,794][__main__][INFO] - Iteration 23 took 18s (22.09% Gen, 72.23% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 13m 24s. Estimated total time: 15h 23m 3s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 46s, 500 more iterations: 2h 33m 50s.
[2025-11-13 08:13:27,797][__main__][INFO] - Starting iteration 23.
[2025-11-13 08:13:27,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:13:27,800][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:31,796][__main__][INFO] - Number of regex retries in iteration 23: 0
[2025-11-13 08:13:31,797][__main__][INFO] - agents played in iteration 23 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:13:32,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,348][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:32,390][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:32,390][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:33,140][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:13:33,437][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:13:33,764][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:13:34,089][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:13:34,416][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:13:34,741][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:13:35,066][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:13:35,394][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:13:35,719][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:13:36,044][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:13:36,373][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:13:36,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:37,030][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:37,356][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:37,681][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:38,008][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:38,676][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:39,002][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:39,328][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:39,654][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:39,979][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:40,303][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:40,629][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:40,957][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:41,282][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:41,608][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:13:41,935][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:13:42,269][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:13:42,594][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:13:42,919][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:13:43,244][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:13:43,575][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:13:44,312][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:13:45,060][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:13:45,061][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:13:45,063][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:13:46,108][__main__][INFO] - Iteration 24 took 18s (21.83% Gen, 72.45% Train). Generation: 3s, Training: 13s. Estimated remaining time: 15h 5m 31s. Estimated total time: 15h 15m 28s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 30s, 500 more iterations: 2h 32m 34s.
[2025-11-13 08:13:46,110][__main__][INFO] - Starting iteration 24.
[2025-11-13 08:13:46,113][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:13:46,113][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:13:50,149][__main__][INFO] - Number of regex retries in iteration 24: 0
[2025-11-13 08:13:50,150][__main__][INFO] - agents played in iteration 24 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:13:50,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:13:50,727][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:13:50,727][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:13:51,485][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:13:51,781][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:13:52,106][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:13:52,431][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:13:52,756][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:13:53,080][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:13:53,406][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:13:53,732][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:13:54,059][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:13:54,383][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:13:54,711][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:13:55,039][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:13:55,367][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:13:55,691][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:13:56,016][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:13:56,342][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:13:56,675][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:13:57,001][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:13:57,326][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:13:57,653][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:13:57,980][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:13:58,306][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:13:58,632][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:13:58,957][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:13:59,283][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:13:59,608][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:13:59,934][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:00,260][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:00,585][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:00,911][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:01,235][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:01,564][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:01,890][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:02,625][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:03,370][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:03,371][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:03,373][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:04,423][__main__][INFO] - Iteration 25 took 18s (22.04% Gen, 72.21% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 5m 18s. Estimated total time: 15h 15m 33s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 31s, 500 more iterations: 2h 32m 35s.
[2025-11-13 08:14:04,425][__main__][INFO] - Starting iteration 25.
[2025-11-13 08:14:04,428][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:04,429][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:08,580][__main__][INFO] - Number of regex retries in iteration 25: 0
[2025-11-13 08:14:08,581][__main__][INFO] - agents played in iteration 25 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:14:09,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,105][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,186][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:09,187][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:09,187][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:09,960][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:14:10,257][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:14:10,584][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:14:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:14:11,237][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:14:11,562][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:14:11,888][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:14:12,217][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:14:12,545][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:14:12,880][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:14:13,212][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:14:13,541][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:14:13,870][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:14:14,198][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:14:14,528][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:14:14,856][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:14:15,184][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:14:15,512][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:14:15,840][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:14:16,169][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:14:16,502][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:14:16,835][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:14:17,168][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:14:17,503][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:14:17,826][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:14:18,156][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:14:18,482][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:18,810][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:19,136][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:19,462][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:19,787][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:20,117][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:20,441][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:21,197][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:21,985][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:21,987][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:21,988][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:23,228][__main__][INFO] - Iteration 26 took 18s (22.08% Gen, 71.32% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 29m 26s. Estimated total time: 15h 40m 1s. Time estimates for 10 more iterations: 3m 8s, 100 more iterations: 31m 20s, 500 more iterations: 2h 36m 40s.
[2025-11-13 08:14:23,230][__main__][INFO] - Starting iteration 26.
[2025-11-13 08:14:23,233][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:23,233][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:27,314][__main__][INFO] - Number of regex retries in iteration 26: 0
[2025-11-13 08:14:27,315][__main__][INFO] - agents played in iteration 26 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:14:27,784][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:27,909][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:27,909][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:28,682][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:14:28,979][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:14:29,304][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:14:29,630][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:14:29,961][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:14:30,289][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:14:30,618][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:14:30,955][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:14:31,280][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:14:31,612][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:14:31,941][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:14:32,266][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:14:32,595][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:14:32,922][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:14:33,251][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:14:33,579][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:14:33,906][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:14:34,234][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:14:34,559][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:14:34,888][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:14:35,221][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:14:35,553][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:14:35,881][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:14:36,215][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:14:36,543][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:14:36,871][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:14:37,201][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:37,529][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:37,857][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:38,182][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:38,507][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:38,834][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:39,161][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:39,916][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:40,673][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:40,678][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:40,679][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:14:41,773][__main__][INFO] - Iteration 27 took 18s (22.01% Gen, 72.08% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 16m 11s. Estimated total time: 15h 27m 4s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 54s, 500 more iterations: 2h 34m 30s.
[2025-11-13 08:14:41,775][__main__][INFO] - Starting iteration 27.
[2025-11-13 08:14:41,778][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:14:41,779][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:14:45,836][__main__][INFO] - Number of regex retries in iteration 27: 0
[2025-11-13 08:14:45,837][__main__][INFO] - agents played in iteration 27 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:14:46,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:46,340][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:46,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:46,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:14:46,423][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:14:46,423][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:14:47,158][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:14:47,453][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:14:47,778][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:14:48,113][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:14:48,442][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:14:48,773][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:14:49,102][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:14:49,430][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:14:49,756][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:14:50,089][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:14:50,426][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:14:50,750][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:14:51,079][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:14:51,406][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:14:51,734][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:14:52,059][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:14:52,387][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:14:52,712][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:14:53,038][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:14:53,367][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:14:53,696][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:14:54,029][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:14:54,354][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:14:54,686][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:14:55,014][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:14:55,346][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:14:55,672][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:14:56,012][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:14:56,337][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:14:56,662][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:14:56,991][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:14:57,321][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:14:57,647][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:14:58,375][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:14:59,114][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:14:59,115][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:14:59,117][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:00,136][__main__][INFO] - Iteration 28 took 18s (22.10% Gen, 72.33% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 6m 46s. Estimated total time: 15h 17m 57s. Time estimates for 10 more iterations: 3m 3s, 100 more iterations: 30m 35s, 500 more iterations: 2h 32m 59s.
[2025-11-13 08:15:00,139][__main__][INFO] - Starting iteration 28.
[2025-11-13 08:15:00,142][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:15:00,142][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:04,208][__main__][INFO] - Number of regex retries in iteration 28: 0
[2025-11-13 08:15:04,209][__main__][INFO] - agents played in iteration 28 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:15:04,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,758][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,798][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:04,799][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:04,799][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:05,563][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:15:05,860][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:15:06,187][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:15:06,519][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:15:06,851][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:15:07,180][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:15:07,509][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:15:07,833][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:15:08,166][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:15:08,492][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:15:08,824][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:15:09,152][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:15:09,482][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:15:09,813][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:15:10,143][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:15:10,475][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:15:10,804][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:15:11,136][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:15:11,463][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:15:11,788][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:15:12,113][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:15:12,439][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:15:12,764][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:15:13,089][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:15:13,414][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:15:13,739][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:15:14,067][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:15:14,395][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:15:14,720][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:15:15,048][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:15:15,373][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:15:15,702][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:15:16,030][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:15:16,773][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:15:17,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:17,530][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:17,531][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:18,609][__main__][INFO] - Iteration 29 took 18s (22.02% Gen, 72.14% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 11m 55s. Estimated total time: 15h 23m 24s. Time estimates for 10 more iterations: 3m 4s, 100 more iterations: 30m 46s, 500 more iterations: 2h 33m 54s.
[2025-11-13 08:15:18,611][__main__][INFO] - Starting iteration 29.
[2025-11-13 08:15:18,614][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:15:18,615][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:22,708][__main__][INFO] - Number of regex retries in iteration 29: 0
[2025-11-13 08:15:22,709][__main__][INFO] - agents played in iteration 29 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:15:23,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:23,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:23,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:23,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:23,302][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:23,302][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:24,040][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:15:24,336][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:15:24,666][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:15:25,005][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:15:25,333][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:15:25,666][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:15:25,994][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:15:26,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:15:26,649][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:15:26,977][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:15:27,305][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:15:27,633][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:15:27,959][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:15:28,286][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:15:28,614][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:15:28,939][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:15:29,272][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:15:29,601][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:15:29,930][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:15:30,257][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:15:30,581][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:15:30,913][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:15:31,238][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:15:31,567][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:15:31,900][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:15:32,229][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:15:32,558][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:15:32,888][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:15:33,213][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:15:33,538][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:15:33,863][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:15:34,190][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:15:34,516][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:15:35,269][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:15:36,035][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:36,036][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:36,038][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:37,176][__main__][INFO] - Iteration 30 took 18s (22.05% Gen, 71.80% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 16m 22s. Estimated total time: 15h 28m 10s. Time estimates for 10 more iterations: 3m 5s, 100 more iterations: 30m 56s, 500 more iterations: 2h 34m 41s.
[2025-11-13 08:15:37,178][__main__][INFO] - Starting iteration 30.
[2025-11-13 08:15:37,181][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 2 and human policies 1.
[2025-11-13 08:15:37,182][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:15:41,233][__main__][INFO] - Number of regex retries in iteration 30: 0
[2025-11-13 08:15:41,234][__main__][INFO] - agents played in iteration 30 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:15:41,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,852][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:15:41,852][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:15:41,853][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:15:42,585][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:15:42,882][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:15:43,208][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:15:43,534][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:15:43,860][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:15:44,185][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:15:44,510][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:15:44,836][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:15:45,161][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:15:45,486][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:15:45,815][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:15:46,141][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:15:46,465][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:15:46,791][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:15:47,115][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:15:47,442][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:15:47,767][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:15:48,094][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:15:48,421][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:15:48,749][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:15:49,074][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:15:49,401][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:15:49,733][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:15:50,061][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:15:50,386][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:15:50,712][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:15:51,037][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:15:51,362][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:15:51,687][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:15:52,013][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:15:52,340][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:15:52,667][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:15:52,993][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:15:53,755][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:15:54,537][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:15:54,538][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:15:54,540][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:15:56,630][__main__][INFO] - Iteration 31 took 19s (20.83% Gen, 68.42% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 0m 21s. Estimated total time: 16h 12m 28s. Time estimates for 10 more iterations: 3m 14s, 100 more iterations: 32m 24s, 500 more iterations: 2h 42m 4s.
[2025-11-13 08:15:56,632][__main__][INFO] - Starting iteration 31.
[2025-11-13 08:15:56,635][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:15:56,636][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:01,171][__main__][INFO] - Number of regex retries in iteration 31: 0
[2025-11-13 08:16:01,172][__main__][INFO] - agents played in iteration 31 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:16:01,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:01,764][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:01,764][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:02,515][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:02,917][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:03,245][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:03,570][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:16:03,896][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:16:04,223][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:16:04,552][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:16:04,877][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:16:05,202][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:16:05,531][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:16:05,853][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:16:06,178][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:16:06,504][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:16:06,832][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:16:07,153][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:16:07,479][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:16:07,807][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:16:08,139][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:16:08,459][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:16:08,786][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:16:09,112][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:16:09,437][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:16:09,763][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:16:10,091][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:16:10,419][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:16:10,744][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:16:11,072][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:16:11,400][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:16:11,728][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:16:12,055][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:16:12,383][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:16:12,711][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:16:13,045][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:16:13,781][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:16:14,541][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:14,543][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:14,545][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:15,743][__main__][INFO] - Iteration 32 took 19s (23.74% Gen, 69.99% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 42m 57s. Estimated total time: 15h 55m 24s. Time estimates for 10 more iterations: 3m 11s, 100 more iterations: 31m 50s, 500 more iterations: 2h 39m 14s.
[2025-11-13 08:16:15,744][__main__][INFO] - Starting iteration 32.
[2025-11-13 08:16:15,747][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:16:15,748][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:20,148][__main__][INFO] - Number of regex retries in iteration 32: 0
[2025-11-13 08:16:20,149][__main__][INFO] - agents played in iteration 32 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:16:20,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,722][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:20,764][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:20,764][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:21,532][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:21,830][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:22,156][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:22,483][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:16:22,815][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:16:23,142][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:16:23,467][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:16:23,792][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:16:24,126][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:16:24,459][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:16:24,786][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:16:25,112][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:16:25,447][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:16:25,772][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:16:26,097][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:16:26,423][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:16:26,753][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:16:27,082][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:16:27,408][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:16:27,734][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:16:28,059][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:16:28,388][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:16:28,714][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:16:29,039][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:16:29,364][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:16:29,690][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:16:30,015][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:16:30,340][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:16:30,666][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:16:30,991][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:16:31,316][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:16:31,645][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:16:31,975][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:16:32,716][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:16:33,488][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:33,489][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:33,491][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:34,447][__main__][INFO] - Iteration 33 took 18s (23.53% Gen, 71.35% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 22m 17s. Estimated total time: 15h 35m 2s. Time estimates for 10 more iterations: 3m 7s, 100 more iterations: 31m 10s, 500 more iterations: 2h 35m 50s.
[2025-11-13 08:16:34,450][__main__][INFO] - Starting iteration 33.
[2025-11-13 08:16:34,453][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:16:34,453][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:38,893][__main__][INFO] - Number of regex retries in iteration 33: 0
[2025-11-13 08:16:38,893][__main__][INFO] - agents played in iteration 33 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:16:39,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,479][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:39,480][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:39,480][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:40,252][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:40,549][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:40,874][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:41,199][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:16:41,531][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:16:41,856][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:16:42,181][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:16:42,505][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:16:42,830][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:16:43,155][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:16:43,481][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:16:43,805][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:16:44,131][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:16:44,459][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:16:44,784][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:16:45,110][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:16:45,438][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:16:45,765][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:16:46,091][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:16:46,417][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:16:46,743][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:16:47,068][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:16:47,394][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:16:47,724][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:16:48,052][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:16:48,378][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:16:48,704][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:16:49,032][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:16:49,358][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:16:49,683][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:16:50,009][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:16:50,335][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:16:50,662][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:16:51,414][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:16:52,191][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:16:52,193][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:16:52,195][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:16:53,181][__main__][INFO] - Iteration 34 took 18s (23.70% Gen, 71.02% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 23m 22s. Estimated total time: 15h 36m 26s. Time estimates for 10 more iterations: 3m 7s, 100 more iterations: 31m 12s, 500 more iterations: 2h 36m 4s.
[2025-11-13 08:16:53,183][__main__][INFO] - Starting iteration 34.
[2025-11-13 08:16:53,185][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:16:53,186][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:16:57,652][__main__][INFO] - Number of regex retries in iteration 34: 0
[2025-11-13 08:16:57,653][__main__][INFO] - agents played in iteration 34 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:16:58,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,231][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,272][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:16:58,272][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:16:58,272][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:16:59,039][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:16:59,338][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:16:59,660][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:16:59,993][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:00,321][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:00,650][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:00,970][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:01,295][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:01,620][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:01,951][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:02,272][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:02,597][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:02,924][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:17:03,259][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:17:03,593][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:17:03,918][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:17:04,246][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:17:04,574][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:17:04,905][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:17:05,234][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:17:05,560][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:17:05,885][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:17:06,223][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:17:06,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:17:06,880][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:17:07,204][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:17:07,543][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:17:07,868][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:17:08,196][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:17:08,520][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:17:08,849][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:17:09,174][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:17:09,503][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:17:10,264][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:17:11,009][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:11,011][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:11,013][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:12,016][__main__][INFO] - Iteration 35 took 18s (23.72% Gen, 70.94% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 28m 12s. Estimated total time: 15h 41m 35s. Time estimates for 10 more iterations: 3m 8s, 100 more iterations: 31m 23s, 500 more iterations: 2h 36m 55s.
[2025-11-13 08:17:12,019][__main__][INFO] - Starting iteration 35.
[2025-11-13 08:17:12,021][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:17:12,022][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:16,565][__main__][INFO] - Number of regex retries in iteration 35: 0
[2025-11-13 08:17:16,566][__main__][INFO] - agents played in iteration 35 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:17:17,042][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,165][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:17,166][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:17,166][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:17,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:17:18,258][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:17:18,589][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:17:18,921][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:19,252][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:19,572][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:19,898][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:20,226][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:20,556][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:20,884][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:21,214][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:21,542][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:21,872][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:17:22,198][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:17:22,525][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:17:22,850][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:17:23,178][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:17:23,519][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:17:23,847][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:17:24,175][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:17:24,502][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:17:24,841][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:17:25,166][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:17:25,492][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:17:25,818][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:17:26,150][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:17:26,478][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:17:26,807][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:17:27,136][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:17:27,465][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:17:27,794][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:17:28,118][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:17:28,450][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:17:29,220][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:17:30,002][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:30,004][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:30,005][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:30,996][__main__][INFO] - Iteration 36 took 18s (23.94% Gen, 70.83% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 35m 4s. Estimated total time: 15h 48m 46s. Time estimates for 10 more iterations: 3m 9s, 100 more iterations: 31m 37s, 500 more iterations: 2h 38m 7s.
[2025-11-13 08:17:30,998][__main__][INFO] - Starting iteration 36.
[2025-11-13 08:17:31,001][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:17:31,001][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:35,590][__main__][INFO] - Number of regex retries in iteration 36: 0
[2025-11-13 08:17:35,591][__main__][INFO] - agents played in iteration 36 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:17:36,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:36,178][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:36,178][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:36,943][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:17:37,239][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:17:37,567][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:17:37,893][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:38,219][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:38,545][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:38,871][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:39,208][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:39,533][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:39,862][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:40,187][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:40,519][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:40,848][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:17:41,177][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:17:41,504][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:17:41,832][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:17:42,159][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:17:42,487][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:17:42,813][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:17:43,140][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:17:43,466][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:17:43,792][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:17:44,118][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:17:44,444][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:17:44,769][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:17:45,094][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:17:45,421][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:17:45,752][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:17:46,073][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:17:46,398][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:17:46,723][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:17:47,054][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:17:47,378][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:17:48,141][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:17:48,932][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:17:48,934][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:17:48,935][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:17:50,257][__main__][INFO] - Iteration 37 took 19s (23.83% Gen, 69.29% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 48m 50s. Estimated total time: 16h 2m 51s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 5s, 500 more iterations: 2h 40m 28s.
[2025-11-13 08:17:50,259][__main__][INFO] - Starting iteration 37.
[2025-11-13 08:17:50,262][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:17:50,263][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:17:54,712][__main__][INFO] - Number of regex retries in iteration 37: 0
[2025-11-13 08:17:54,712][__main__][INFO] - agents played in iteration 37 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:17:55,173][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:17:55,294][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:17:55,295][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:17:56,028][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:17:56,335][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:17:56,661][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:17:56,988][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:17:57,314][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:17:57,652][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:17:57,984][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:17:58,310][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:17:58,639][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:17:58,967][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:17:59,294][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:17:59,620][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:17:59,948][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:00,278][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:00,611][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:00,940][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:01,266][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:01,591][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:01,919][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:02,245][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:02,570][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:02,896][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:18:03,222][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:18:03,547][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:18:03,875][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:18:04,203][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:18:04,530][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:18:04,856][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:18:05,185][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:18:05,516][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:18:05,842][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:18:06,173][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:18:06,499][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:18:07,249][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:18:08,022][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:08,023][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:08,025][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:09,053][__main__][INFO] - Iteration 38 took 18s (23.68% Gen, 70.84% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 25m 14s. Estimated total time: 15h 39m 35s. Time estimates for 10 more iterations: 3m 7s, 100 more iterations: 31m 19s, 500 more iterations: 2h 36m 35s.
[2025-11-13 08:18:09,055][__main__][INFO] - Starting iteration 38.
[2025-11-13 08:18:09,058][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:18:09,058][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:13,665][__main__][INFO] - Number of regex retries in iteration 38: 0
[2025-11-13 08:18:13,666][__main__][INFO] - agents played in iteration 38 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:18:14,128][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,168][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:14,249][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:18:14,249][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:18:15,005][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:18:15,302][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:18:15,628][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:18:15,954][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:18:16,278][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:18:16,603][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:18:16,928][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:18:17,253][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:18:17,578][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:18:17,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:18:18,229][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:18:18,556][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:18:18,881][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:19,207][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:19,533][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:19,859][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:20,186][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:20,512][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:20,837][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:21,162][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:21,489][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:21,814][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:18:22,141][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:18:22,466][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:18:22,792][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:18:23,117][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:18:23,445][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:18:23,771][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:18:24,098][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:18:24,424][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:18:24,749][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:18:25,078][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:18:25,406][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:18:26,156][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:18:26,910][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:26,911][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:26,913][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:27,982][__main__][INFO] - Iteration 39 took 18s (24.35% Gen, 70.00% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 31m 37s. Estimated total time: 15h 46m 16s. Time estimates for 10 more iterations: 3m 9s, 100 more iterations: 31m 32s, 500 more iterations: 2h 37m 42s.
[2025-11-13 08:18:27,984][__main__][INFO] - Starting iteration 39.
[2025-11-13 08:18:27,988][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:18:27,988][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:32,529][__main__][INFO] - Number of regex retries in iteration 39: 0
[2025-11-13 08:18:32,530][__main__][INFO] - agents played in iteration 39 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:18:33,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,091][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:33,132][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:18:33,132][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:18:33,900][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:18:34,203][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:18:34,524][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:18:34,848][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:18:35,173][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:18:35,505][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:18:35,831][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:18:36,157][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:18:36,483][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:18:36,810][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:18:37,139][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:18:37,465][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:18:37,791][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:38,117][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:38,443][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:38,769][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:39,094][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:39,421][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:39,747][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:40,072][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:40,398][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:40,723][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:18:41,051][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:18:41,377][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:18:41,705][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:18:42,032][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:18:42,363][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:18:42,692][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:18:43,017][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:18:43,343][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:18:43,681][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:18:44,006][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:18:44,335][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:18:45,077][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:18:45,833][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:18:45,835][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:18:45,837][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:18:46,868][__main__][INFO] - Iteration 40 took 18s (24.05% Gen, 70.47% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 29m 9s. Estimated total time: 15h 44m 7s. Time estimates for 10 more iterations: 3m 8s, 100 more iterations: 31m 28s, 500 more iterations: 2h 37m 21s.
[2025-11-13 08:18:46,871][__main__][INFO] - Starting iteration 40.
[2025-11-13 08:18:46,875][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 3 and human policies 1.
[2025-11-13 08:18:46,875][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:18:51,519][__main__][INFO] - Number of regex retries in iteration 40: 0
[2025-11-13 08:18:51,519][__main__][INFO] - agents played in iteration 40 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:18:51,998][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,079][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,120][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:18:52,121][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:18:52,121][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:18:52,905][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:18:53,204][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:18:53,529][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:18:53,857][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:18:54,182][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:18:54,509][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:18:54,834][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:18:55,159][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:18:55,485][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:18:55,810][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:18:56,136][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:18:56,461][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:18:56,786][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:18:57,111][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:18:57,441][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:18:57,766][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:18:58,093][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:18:58,419][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:18:58,744][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:18:59,069][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:18:59,396][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:18:59,721][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:00,046][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:00,372][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:00,698][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:19:01,022][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:19:01,349][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:19:01,675][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:19:02,003][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:19:02,329][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:19:02,655][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:19:02,980][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:19:03,309][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:19:04,058][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:19:04,834][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:04,835][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:04,837][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:07,421][__main__][INFO] - Iteration 41 took 20s (22.60% Gen, 64.82% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 52m 2s. Estimated total time: 17h 7m 21s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 14s, 500 more iterations: 2h 51m 13s.
[2025-11-13 08:19:07,423][__main__][INFO] - Starting iteration 41.
[2025-11-13 08:19:07,426][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:19:07,426][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:19:12,400][__main__][INFO] - Number of regex retries in iteration 41: 0
[2025-11-13 08:19:12,401][__main__][INFO] - agents played in iteration 41 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:19:12,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:12,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:12,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:13,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:13,004][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:13,004][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:13,735][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:19:14,031][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:19:14,356][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:19:14,681][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:19:15,006][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:19:15,330][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:19:15,658][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:19:15,982][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:19:16,309][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:19:16,638][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:19:16,966][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:19:17,292][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:19:17,617][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:19:17,942][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:19:18,267][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:19:18,593][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:19:18,918][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:19:19,243][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:19:19,568][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:19:19,894][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:19:20,222][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:19:20,549][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:20,875][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:21,207][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:21,540][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:19:21,868][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:19:22,195][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:19:22,519][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:19:22,845][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:19:23,171][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:19:23,497][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:19:23,823][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:19:24,151][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:19:24,894][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:19:25,646][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:25,648][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:25,650][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:26,668][__main__][INFO] - Iteration 42 took 19s (25.85% Gen, 68.85% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 46m 30s. Estimated total time: 16h 2m 8s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 4s, 500 more iterations: 2h 40m 21s.
[2025-11-13 08:19:26,670][__main__][INFO] - Starting iteration 42.
[2025-11-13 08:19:26,673][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:19:26,673][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:19:31,446][__main__][INFO] - Number of regex retries in iteration 42: 0
[2025-11-13 08:19:31,446][__main__][INFO] - agents played in iteration 42 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:19:31,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:31,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:31,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:32,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:32,027][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:32,028][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:32,774][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:19:33,070][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:19:33,394][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:19:33,722][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:19:34,045][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:19:34,372][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:19:34,697][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:19:35,022][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:19:35,347][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:19:35,673][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:19:35,999][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:19:36,323][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:19:36,647][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:19:36,973][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:19:37,305][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:19:37,627][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:19:37,952][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:19:38,276][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:19:38,607][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:19:38,926][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:19:39,251][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:19:39,576][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:39,901][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:40,226][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:40,550][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:19:40,875][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:19:41,201][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:19:41,526][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:19:41,851][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:19:42,176][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:19:42,507][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:19:42,826][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:19:43,153][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:19:43,904][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:19:44,657][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:19:44,658][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:19:44,660][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:19:45,734][__main__][INFO] - Iteration 43 took 19s (25.04% Gen, 69.32% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 37m 9s. Estimated total time: 15h 53m 5s. Time estimates for 10 more iterations: 3m 10s, 100 more iterations: 31m 46s, 500 more iterations: 2h 38m 50s.
[2025-11-13 08:19:45,736][__main__][INFO] - Starting iteration 43.
[2025-11-13 08:19:45,740][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:19:45,740][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:19:50,769][__main__][INFO] - Number of regex retries in iteration 43: 0
[2025-11-13 08:19:50,770][__main__][INFO] - agents played in iteration 43 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:19:51,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,344][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:19:51,385][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:19:51,385][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:19:52,165][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:19:52,462][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:19:52,788][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:19:53,115][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:19:53,441][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:19:53,767][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:19:54,093][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:19:54,419][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:19:54,748][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:19:55,076][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:19:55,402][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:19:55,730][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:19:56,056][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:19:56,380][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:19:56,706][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:19:57,031][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:19:57,361][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:19:57,686][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:19:58,011][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:19:58,336][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:19:58,662][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:19:58,987][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:19:59,312][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:19:59,637][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:19:59,963][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:00,288][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:00,613][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:00,940][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:01,265][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:01,591][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:01,917][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:02,243][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:20:02,570][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:20:03,387][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:20:04,160][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:04,162][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:04,164][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:05,178][__main__][INFO] - Iteration 44 took 19s (25.87% Gen, 68.91% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 55m 40s. Estimated total time: 16h 11m 56s. Time estimates for 10 more iterations: 3m 14s, 100 more iterations: 32m 23s, 500 more iterations: 2h 41m 59s.
[2025-11-13 08:20:05,180][__main__][INFO] - Starting iteration 44.
[2025-11-13 08:20:05,183][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:20:05,184][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:10,113][__main__][INFO] - Number of regex retries in iteration 44: 0
[2025-11-13 08:20:10,114][__main__][INFO] - agents played in iteration 44 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:20:10,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,627][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:10,709][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:10,709][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:11,480][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:20:11,783][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:20:12,108][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:20:12,434][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:20:12,765][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:20:13,094][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:20:13,419][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:20:13,747][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:20:14,072][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:20:14,397][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:20:14,722][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:20:15,050][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:20:15,376][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:20:15,702][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:20:16,029][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:20:16,358][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:20:16,686][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:20:17,011][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:20:17,335][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:20:17,661][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:20:17,992][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:20:18,321][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:20:18,647][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:20:18,972][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:20:19,298][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:19,625][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:19,951][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:20,276][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:20,606][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:20,928][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:21,254][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:21,579][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:20:21,906][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:20:22,669][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:20:23,455][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:23,457][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:23,458][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:24,449][__main__][INFO] - Iteration 45 took 19s (25.59% Gen, 69.26% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 46m 44s. Estimated total time: 16h 3m 19s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 6s, 500 more iterations: 2h 40m 33s.
[2025-11-13 08:20:24,451][__main__][INFO] - Starting iteration 45.
[2025-11-13 08:20:24,454][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:20:24,455][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:29,326][__main__][INFO] - Number of regex retries in iteration 45: 0
[2025-11-13 08:20:29,327][__main__][INFO] - agents played in iteration 45 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:20:29,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,849][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,890][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:29,931][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:29,931][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:30,695][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:20:30,993][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:20:31,328][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:20:31,650][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:20:31,976][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:20:32,301][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:20:32,629][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:20:32,958][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:20:33,283][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:20:33,609][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:20:33,934][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:20:34,260][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:20:34,585][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:20:34,912][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:20:35,237][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:20:35,564][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:20:35,892][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:20:36,224][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:20:36,549][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:20:36,874][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:20:37,200][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:20:37,525][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:20:37,853][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:20:38,179][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:20:38,504][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:38,829][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:39,155][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:39,480][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:39,807][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:40,135][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:40,461][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:20:40,786][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:20:41,114][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:20:41,884][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:20:42,631][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:20:42,632][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:20:42,634][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:20:43,654][__main__][INFO] - Iteration 46 took 19s (25.37% Gen, 69.30% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 43m 7s. Estimated total time: 16h 0m 2s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 0s, 500 more iterations: 2h 40m 0s.
[2025-11-13 08:20:43,657][__main__][INFO] - Starting iteration 46.
[2025-11-13 08:20:43,660][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:20:43,660][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:20:48,605][__main__][INFO] - Number of regex retries in iteration 46: 0
[2025-11-13 08:20:48,605][__main__][INFO] - agents played in iteration 46 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:20:49,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,109][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,191][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:20:49,192][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:20:49,192][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:20:49,967][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:20:50,265][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:20:50,590][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:20:50,920][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:20:51,243][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:20:51,570][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:20:51,898][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:20:52,227][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:20:52,550][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:20:52,875][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:20:53,201][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:20:53,530][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:20:53,853][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:20:54,178][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:20:54,504][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:20:54,829][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:20:55,154][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:20:55,481][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:20:55,806][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:20:56,133][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:20:56,459][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:20:56,784][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:20:57,109][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:20:57,436][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:20:57,762][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:20:58,087][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:20:58,413][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:20:58,739][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:20:59,071][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:20:59,398][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:20:59,722][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:00,047][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:00,375][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:01,128][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:01,885][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:01,886][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:01,888][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:02,907][__main__][INFO] - Iteration 47 took 19s (25.69% Gen, 69.01% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 45m 9s. Estimated total time: 16h 2m 23s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 4s, 500 more iterations: 2h 40m 23s.
[2025-11-13 08:21:02,909][__main__][INFO] - Starting iteration 47.
[2025-11-13 08:21:02,913][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:21:02,913][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:07,927][__main__][INFO] - Number of regex retries in iteration 47: 0
[2025-11-13 08:21:07,928][__main__][INFO] - agents played in iteration 47 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:21:08,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:08,513][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:08,513][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:09,268][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:21:09,564][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:21:09,889][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:21:10,215][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:21:10,540][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:21:10,865][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:21:11,190][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:21:11,514][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:21:11,840][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:21:12,167][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:21:12,492][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:21:12,821][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:21:13,146][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:21:13,473][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:21:13,799][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:21:14,128][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:21:14,455][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:21:14,780][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:21:15,111][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:21:15,437][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:21:15,764][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:21:16,094][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:21:16,419][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:21:16,745][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:21:17,069][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:21:17,402][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:21:17,727][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:21:18,054][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:21:18,381][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:21:18,720][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:21:19,046][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:19,374][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:19,703][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:20,449][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:21,210][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:21,211][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:21,213][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:22,216][__main__][INFO] - Iteration 48 took 19s (25.98% Gen, 68.82% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 47m 40s. Estimated total time: 16h 5m 13s. Time estimates for 10 more iterations: 3m 13s, 100 more iterations: 32m 10s, 500 more iterations: 2h 40m 52s.
[2025-11-13 08:21:22,219][__main__][INFO] - Starting iteration 48.
[2025-11-13 08:21:22,223][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:21:22,223][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:27,146][__main__][INFO] - Number of regex retries in iteration 48: 0
[2025-11-13 08:21:27,146][__main__][INFO] - agents played in iteration 48 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:21:27,618][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:27,759][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:27,759][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:28,552][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:21:28,849][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:21:29,175][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:21:29,502][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:21:29,828][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:21:30,154][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:21:30,480][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:21:30,808][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:21:31,133][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:21:31,460][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:21:31,787][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:21:32,126][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:21:32,451][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:21:32,778][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:21:33,103][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:21:33,433][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:21:33,759][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:21:34,087][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:21:34,413][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:21:34,740][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:21:35,069][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:21:35,395][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:21:35,722][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:21:36,050][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:21:36,378][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:21:36,704][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:21:37,029][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:21:37,354][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:21:37,682][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:21:38,010][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:21:38,338][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:38,667][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:38,995][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:39,753][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:40,504][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:40,505][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:40,507][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:21:41,521][__main__][INFO] - Iteration 49 took 19s (25.51% Gen, 69.23% Train). Generation: 4s, Training: 13s. Estimated remaining time: 15h 47m 5s. Estimated total time: 16h 4m 57s. Time estimates for 10 more iterations: 3m 12s, 100 more iterations: 32m 9s, 500 more iterations: 2h 40m 49s.
[2025-11-13 08:21:41,523][__main__][INFO] - Starting iteration 49.
[2025-11-13 08:21:41,526][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:21:41,527][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:21:46,504][__main__][INFO] - Number of regex retries in iteration 49: 0
[2025-11-13 08:21:46,505][__main__][INFO] - agents played in iteration 49 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:21:46,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,056][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:21:47,098][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:21:47,098][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:21:47,886][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:21:48,182][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:21:48,510][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:21:48,836][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:21:49,162][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:21:49,489][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:21:49,820][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:21:50,147][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:21:50,476][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:21:50,801][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:21:51,127][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:21:51,453][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:21:51,778][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:21:52,104][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:21:52,429][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:21:52,755][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:21:53,084][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:21:53,410][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:21:53,735][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:21:54,061][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:21:54,387][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:21:54,713][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:21:55,041][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:21:55,370][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:21:55,695][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:21:56,020][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:21:56,346][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:21:56,673][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:21:56,998][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:21:57,327][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:21:57,654][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:21:57,982][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:21:58,312][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:21:59,072][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:21:59,838][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:21:59,840][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:21:59,842][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:01,100][__main__][INFO] - Iteration 50 took 19s (25.43% Gen, 68.13% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 0m 31s. Estimated total time: 16h 18m 43s. Time estimates for 10 more iterations: 3m 15s, 100 more iterations: 32m 37s, 500 more iterations: 2h 43m 7s.
[2025-11-13 08:22:01,102][__main__][INFO] - Starting iteration 50.
[2025-11-13 08:22:01,105][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 4 and human policies 1.
[2025-11-13 08:22:01,105][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:06,058][__main__][INFO] - Number of regex retries in iteration 50: 0
[2025-11-13 08:22:06,058][__main__][INFO] - agents played in iteration 50 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:22:06,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:06,653][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:06,653][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:07,410][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:22:07,708][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:22:08,034][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:22:08,361][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:22:08,686][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:22:09,016][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:22:09,342][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:22:09,668][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:22:09,994][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:22:10,321][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:22:10,646][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:22:10,973][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:22:11,298][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:22:11,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:22:11,950][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:22:12,278][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:22:12,600][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:22:12,926][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:22:13,252][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:22:13,581][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:22:13,905][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:22:14,233][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:22:14,560][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:22:14,885][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:22:15,211][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:22:15,537][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:22:15,865][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:22:16,190][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:22:16,516][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:22:16,842][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:22:17,166][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:22:17,493][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:22:17,820][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:22:18,589][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:22:19,377][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:19,380][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:19,382][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:21,377][__main__][INFO] - Iteration 51 took 20s (24.43% Gen, 65.72% Train). Generation: 4s, Training: 13s. Estimated remaining time: 16h 35m 7s. Estimated total time: 16h 53m 40s. Time estimates for 10 more iterations: 3m 22s, 100 more iterations: 33m 47s, 500 more iterations: 2h 48m 56s.
[2025-11-13 08:22:21,380][__main__][INFO] - Starting iteration 51.
[2025-11-13 08:22:21,384][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:22:21,385][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:26,832][__main__][INFO] - Number of regex retries in iteration 51: 0
[2025-11-13 08:22:26,832][__main__][INFO] - agents played in iteration 51 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:22:27,295][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,416][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:27,416][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:27,416][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:28,202][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:22:28,499][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:22:28,826][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:22:29,150][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:22:29,476][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:22:29,802][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:22:30,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:22:30,453][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:22:30,780][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:22:31,104][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:22:31,429][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:22:31,755][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:22:32,084][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:22:32,408][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:22:32,736][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:22:33,061][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:22:33,388][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:22:33,714][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:22:34,040][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:22:34,366][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:22:34,691][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:22:35,018][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:22:35,343][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:22:35,669][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:22:35,996][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:22:36,329][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:22:36,658][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:22:36,985][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:22:37,312][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:22:37,637][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:22:37,964][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:22:38,289][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:22:38,618][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:22:39,393][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:22:40,183][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:40,184][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:40,186][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:22:41,216][__main__][INFO] - Iteration 52 took 19s (27.46% Gen, 67.33% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 12m 47s. Estimated total time: 16h 31m 39s. Time estimates for 10 more iterations: 3m 18s, 100 more iterations: 33m 3s, 500 more iterations: 2h 45m 16s.
[2025-11-13 08:22:41,219][__main__][INFO] - Starting iteration 52.
[2025-11-13 08:22:41,223][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:22:41,223][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:22:46,434][__main__][INFO] - Number of regex retries in iteration 52: 0
[2025-11-13 08:22:46,434][__main__][INFO] - agents played in iteration 52 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:22:46,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:46,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:46,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:47,033][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:22:47,034][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:22:47,034][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:22:47,810][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:22:48,106][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:22:48,434][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:22:48,762][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:22:49,092][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:22:49,425][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:22:49,749][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:22:50,076][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:22:50,402][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:22:50,735][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:22:51,067][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:22:51,396][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:22:51,722][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:22:52,062][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:22:52,395][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:22:52,729][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:22:53,061][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:22:53,387][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:22:53,716][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:22:54,042][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:22:54,374][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:22:54,696][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:22:55,026][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:22:55,351][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:22:55,685][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:22:56,008][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:22:56,334][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:22:56,661][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:22:56,989][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:22:57,315][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:22:57,642][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:22:57,969][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:22:58,297][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:22:59,061][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:22:59,845][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:22:59,846][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:22:59,848][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:00,967][__main__][INFO] - Iteration 53 took 19s (26.39% Gen, 67.94% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 8m 2s. Estimated total time: 16h 27m 14s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 54s, 500 more iterations: 2h 44m 32s.
[2025-11-13 08:23:00,968][__main__][INFO] - Starting iteration 53.
[2025-11-13 08:23:00,971][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:00,972][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:06,131][__main__][INFO] - Number of regex retries in iteration 53: 0
[2025-11-13 08:23:06,131][__main__][INFO] - agents played in iteration 53 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:23:06,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,651][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,741][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:06,741][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:06,742][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:07,530][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:23:07,827][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:23:08,154][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:23:08,480][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:23:08,805][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:23:09,131][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:23:09,456][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:23:09,782][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:23:10,111][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:23:10,439][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:23:10,771][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:23:11,106][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:23:11,433][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:23:11,761][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:23:12,088][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:23:12,414][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:23:12,741][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:23:13,073][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:23:13,401][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:23:13,728][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:23:14,054][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:23:14,383][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:23:14,708][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:23:15,034][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:23:15,360][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:23:15,687][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:23:16,014][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:23:16,340][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:23:16,667][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:23:17,000][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:23:17,326][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:23:17,656][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:23:17,982][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:23:18,745][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:23:19,533][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:19,535][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:19,536][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:20,558][__main__][INFO] - Iteration 54 took 19s (26.34% Gen, 68.44% Train). Generation: 5s, Training: 13s. Estimated remaining time: 15h 59m 51s. Estimated total time: 16h 19m 23s. Time estimates for 10 more iterations: 3m 15s, 100 more iterations: 32m 38s, 500 more iterations: 2h 43m 13s.
[2025-11-13 08:23:20,560][__main__][INFO] - Starting iteration 54.
[2025-11-13 08:23:20,564][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:20,564][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:25,817][__main__][INFO] - Number of regex retries in iteration 54: 0
[2025-11-13 08:23:25,817][__main__][INFO] - agents played in iteration 54 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:23:26,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,379][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:26,420][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:26,420][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:27,189][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:23:27,486][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:23:27,813][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:23:28,138][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:23:28,464][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:23:28,790][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:23:29,116][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:23:29,442][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:23:29,766][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:23:30,095][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:23:30,425][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:23:30,751][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:23:31,077][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:23:31,402][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:23:31,728][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:23:32,054][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:23:32,381][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:23:32,707][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:23:33,033][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:23:33,358][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:23:33,684][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:23:34,010][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:23:34,338][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:23:34,664][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:23:34,990][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:23:35,315][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:23:35,644][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:23:35,974][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:23:36,300][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:23:36,625][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:23:36,961][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:23:37,287][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:23:37,614][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:23:38,383][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:23:39,148][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:39,149][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:39,151][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:40,177][__main__][INFO] - Iteration 55 took 19s (26.78% Gen, 67.98% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 0m 53s. Estimated total time: 16h 20m 44s. Time estimates for 10 more iterations: 3m 16s, 100 more iterations: 32m 41s, 500 more iterations: 2h 43m 27s.
[2025-11-13 08:23:40,179][__main__][INFO] - Starting iteration 55.
[2025-11-13 08:23:40,183][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:40,183][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:23:45,517][__main__][INFO] - Number of regex retries in iteration 55: 0
[2025-11-13 08:23:45,518][__main__][INFO] - agents played in iteration 55 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:23:45,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,072][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:23:46,113][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:23:46,113][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:23:46,890][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:23:47,193][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:23:47,515][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:23:47,841][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:23:48,171][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:23:48,501][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:23:48,823][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:23:49,149][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:23:49,474][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:23:49,802][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:23:50,127][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:23:50,455][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:23:50,783][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:23:51,110][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:23:51,442][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:23:51,768][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:23:52,094][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:23:52,421][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:23:52,746][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:23:53,072][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:23:53,399][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:23:53,725][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:23:54,052][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:23:54,378][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:23:54,704][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:23:55,030][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:23:55,358][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:23:55,684][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:23:56,011][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:23:56,337][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:23:56,667][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:23:56,992][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:23:57,319][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:23:58,075][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:23:58,845][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:23:58,846][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:23:58,848][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:23:59,889][__main__][INFO] - Iteration 56 took 19s (27.07% Gen, 67.64% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 12s. Estimated total time: 16h 25m 23s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 50s, 500 more iterations: 2h 44m 13s.
[2025-11-13 08:23:59,891][__main__][INFO] - Starting iteration 56.
[2025-11-13 08:23:59,894][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:23:59,895][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:05,182][__main__][INFO] - Number of regex retries in iteration 56: 0
[2025-11-13 08:24:05,183][__main__][INFO] - agents played in iteration 56 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:24:05,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,709][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:05,807][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:05,807][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:06,582][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:24:06,879][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:24:07,206][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:24:07,532][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:24:07,859][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:24:08,185][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:24:08,513][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:24:08,838][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:24:09,165][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:24:09,490][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:24:09,816][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:24:10,142][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:24:10,475][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:24:10,803][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:24:11,129][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:24:11,458][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:24:11,783][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:24:12,109][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:24:12,436][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:24:12,762][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:24:13,090][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:24:13,415][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:24:13,744][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:24:14,073][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:24:14,400][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:24:14,727][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:24:15,054][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:24:15,381][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:24:15,707][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:24:16,033][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:24:16,369][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:24:16,695][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:24:17,022][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:24:17,784][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:24:18,550][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:24:18,552][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:24:18,553][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:24:19,615][__main__][INFO] - Iteration 57 took 19s (26.81% Gen, 67.80% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 33s. Estimated total time: 16h 26m 4s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 52s, 500 more iterations: 2h 44m 20s.
[2025-11-13 08:24:19,617][__main__][INFO] - Starting iteration 57.
[2025-11-13 08:24:19,621][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:24:19,622][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:24,896][__main__][INFO] - Number of regex retries in iteration 57: 0
[2025-11-13 08:24:24,897][__main__][INFO] - agents played in iteration 57 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:24:25,366][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,423][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:25,504][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:25,504][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:26,283][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:24:26,581][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:24:26,909][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:24:27,237][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:24:27,564][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:24:27,890][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:24:28,219][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:24:28,546][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:24:28,874][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:24:29,200][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:24:29,525][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:24:29,851][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:24:30,179][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:24:30,507][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:24:30,835][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:24:31,162][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:24:31,488][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:24:31,814][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:24:32,145][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:24:32,471][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:24:32,797][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:24:33,123][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:24:33,453][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:24:33,779][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:24:34,106][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:24:34,432][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:24:34,763][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:24:35,093][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:24:35,422][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:24:35,748][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:24:36,087][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:24:36,414][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:24:36,741][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:24:37,506][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:24:38,280][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:24:38,281][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:24:38,283][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:24:39,307][__main__][INFO] - Iteration 58 took 19s (26.79% Gen, 68.00% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 3m 31s. Estimated total time: 16h 24m 22s. Time estimates for 10 more iterations: 3m 16s, 100 more iterations: 32m 48s, 500 more iterations: 2h 44m 3s.
[2025-11-13 08:24:39,309][__main__][INFO] - Starting iteration 58.
[2025-11-13 08:24:39,314][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:24:39,314][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:24:44,612][__main__][INFO] - Number of regex retries in iteration 58: 0
[2025-11-13 08:24:44,613][__main__][INFO] - agents played in iteration 58 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:24:45,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,186][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,226][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:24:45,226][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:24:45,227][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:24:46,004][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:24:46,307][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:24:46,628][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:24:46,962][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:24:47,291][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:24:47,621][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:24:47,946][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:24:48,272][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:24:48,606][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:24:48,935][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:24:49,261][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:24:49,587][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:24:49,913][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:24:50,239][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:24:50,567][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:24:50,893][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:24:51,219][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:24:51,551][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:24:51,885][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:24:52,215][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:24:52,540][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:24:52,868][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:24:53,207][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:24:53,533][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:24:53,860][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:24:54,186][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:24:54,514][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:24:54,841][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:24:55,167][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:24:55,499][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:24:55,820][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:24:56,151][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:24:56,477][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:24:57,266][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:24:58,031][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:24:58,032][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:24:58,034][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:24:59,046][__main__][INFO] - Iteration 59 took 19s (26.85% Gen, 68.02% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 5m 30s. Estimated total time: 16h 26m 41s. Time estimates for 10 more iterations: 3m 17s, 100 more iterations: 32m 53s, 500 more iterations: 2h 44m 26s.
[2025-11-13 08:24:59,049][__main__][INFO] - Starting iteration 59.
[2025-11-13 08:24:59,053][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:24:59,053][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:04,356][__main__][INFO] - Number of regex retries in iteration 59: 0
[2025-11-13 08:25:04,357][__main__][INFO] - agents played in iteration 59 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:25:04,830][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:04,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:04,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:04,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:04,952][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:04,952][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:05,736][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:25:06,034][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:25:06,361][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:25:06,690][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:25:07,016][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:25:07,342][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:25:07,669][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:25:07,995][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:25:08,321][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:25:08,649][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:25:08,976][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:25:09,305][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:25:09,632][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:25:09,958][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:25:10,285][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:25:10,613][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:25:10,941][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:25:11,269][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:25:11,595][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:25:11,922][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:25:12,247][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:25:12,574][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:25:12,903][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:25:13,230][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:25:13,556][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:25:13,882][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:25:14,209][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:25:14,536][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:25:14,863][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:25:15,190][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:25:15,517][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:25:15,843][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:25:16,169][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:25:16,944][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:25:17,710][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:17,712][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:17,715][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:18,933][__main__][INFO] - Iteration 60 took 19s (26.67% Gen, 67.20% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 12m 33s. Estimated total time: 16h 34m 3s. Time estimates for 10 more iterations: 3m 18s, 100 more iterations: 33m 8s, 500 more iterations: 2h 45m 40s.
[2025-11-13 08:25:18,935][__main__][INFO] - Starting iteration 60.
[2025-11-13 08:25:18,938][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 5 and human policies 1.
[2025-11-13 08:25:18,939][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:24,166][__main__][INFO] - Number of regex retries in iteration 60: 0
[2025-11-13 08:25:24,167][__main__][INFO] - agents played in iteration 60 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:25:24,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:24,773][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:24,773][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:25,538][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:25:25,836][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:25:26,163][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:25:26,488][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:25:26,819][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:25:27,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:25:27,472][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:25:27,798][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:25:28,124][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:25:28,450][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:25:28,776][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:25:29,102][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:25:29,432][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:25:29,764][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:25:30,091][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:25:30,417][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:25:30,743][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:25:31,076][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:25:31,404][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:25:31,734][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:25:32,063][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:25:32,398][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:25:32,724][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:25:33,053][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:25:33,381][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:25:33,710][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:25:34,036][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:25:34,362][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:25:34,689][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:25:35,019][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:25:35,348][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:25:35,676][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:25:36,007][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:25:36,770][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:25:37,535][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:37,536][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:37,538][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:39,603][__main__][INFO] - Iteration 61 took 20s (25.30% Gen, 64.70% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 51m 27s. Estimated total time: 17h 13m 17s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 26s, 500 more iterations: 2h 52m 12s.
[2025-11-13 08:25:39,605][__main__][INFO] - Starting iteration 61.
[2025-11-13 08:25:39,608][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:25:39,609][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:25:45,179][__main__][INFO] - Number of regex retries in iteration 61: 0
[2025-11-13 08:25:45,180][__main__][INFO] - agents played in iteration 61 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:25:45,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:25:45,787][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:25:45,787][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:25:46,544][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:25:46,840][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:25:47,164][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:25:47,489][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:25:47,817][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:25:48,143][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:25:48,469][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:25:48,795][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:25:49,126][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:25:49,455][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:25:49,784][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:25:50,109][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:25:50,435][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:25:50,761][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:25:51,088][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:25:51,414][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:25:51,740][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:25:52,067][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:25:52,394][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:25:52,721][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:25:53,048][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:25:53,375][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:25:53,700][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:25:54,027][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:25:54,353][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:25:54,682][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:25:55,009][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:25:55,336][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:25:55,665][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:25:55,994][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:25:56,320][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:25:56,646][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:25:56,972][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:25:57,735][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:25:58,471][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:25:58,473][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:25:58,475][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:25:59,556][__main__][INFO] - Iteration 62 took 19s (27.93% Gen, 66.65% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 15m 14s. Estimated total time: 16h 37m 25s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 14s, 500 more iterations: 2h 46m 14s.
[2025-11-13 08:25:59,558][__main__][INFO] - Starting iteration 62.
[2025-11-13 08:25:59,561][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:25:59,562][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:05,086][__main__][INFO] - Number of regex retries in iteration 62: 0
[2025-11-13 08:26:05,087][__main__][INFO] - agents played in iteration 62 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:26:05,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,692][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:05,692][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:05,693][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:06,486][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:26:06,784][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:26:07,111][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:26:07,439][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:26:07,771][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:26:08,110][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:26:08,438][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:26:08,766][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:26:09,095][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:26:09,423][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:26:09,748][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:26:10,074][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:26:10,402][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:26:10,727][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:26:11,054][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:26:11,380][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:26:11,707][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:26:12,034][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:26:12,360][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:26:12,688][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:26:13,014][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:26:13,343][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:26:13,669][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:26:13,996][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:26:14,322][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:26:14,648][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:26:14,975][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:26:15,303][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:26:15,630][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:26:15,958][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:26:16,284][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:26:16,611][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:26:16,937][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:26:17,714][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:26:18,496][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:18,498][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:18,499][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:19,560][__main__][INFO] - Iteration 63 took 20s (27.63% Gen, 67.06% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 17m 29s. Estimated total time: 16h 40m 0s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 20s, 500 more iterations: 2h 46m 40s.
[2025-11-13 08:26:19,562][__main__][INFO] - Starting iteration 63.
[2025-11-13 08:26:19,565][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:26:19,565][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:25,119][__main__][INFO] - Number of regex retries in iteration 63: 0
[2025-11-13 08:26:25,120][__main__][INFO] - agents played in iteration 63 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:26:25,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,626][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,705][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:25,706][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:25,706][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:26,475][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:26:26,773][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:26:27,099][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:26:27,426][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:26:27,751][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:26:28,078][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:26:28,406][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:26:28,736][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:26:29,063][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:26:29,388][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:26:29,717][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:26:30,047][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:26:30,374][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:26:30,700][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:26:31,025][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:26:31,350][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:26:31,690][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:26:32,016][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:26:32,342][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:26:32,669][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:26:33,003][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:26:33,329][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:26:33,655][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:26:33,984][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:26:34,322][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:26:34,649][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:26:34,974][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:26:35,300][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:26:35,627][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:26:35,953][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:26:36,279][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:26:36,607][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:26:36,935][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:26:37,686][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:26:38,472][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:38,474][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:38,476][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:39,520][__main__][INFO] - Iteration 64 took 19s (27.83% Gen, 66.92% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 14m 58s. Estimated total time: 16h 37m 49s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 15s, 500 more iterations: 2h 46m 18s.
[2025-11-13 08:26:39,523][__main__][INFO] - Starting iteration 64.
[2025-11-13 08:26:39,526][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:26:39,526][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:26:45,065][__main__][INFO] - Number of regex retries in iteration 64: 0
[2025-11-13 08:26:45,066][__main__][INFO] - agents played in iteration 64 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:26:45,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:26:45,661][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:26:45,662][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:26:46,439][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:26:46,735][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:26:47,061][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:26:47,389][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:26:47,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:26:48,053][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:26:48,381][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:26:48,706][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:26:49,031][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:26:49,357][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:26:49,685][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:26:50,009][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:26:50,335][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:26:50,663][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:26:50,995][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:26:51,321][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:26:51,647][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:26:51,973][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:26:52,299][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:26:52,625][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:26:52,954][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:26:53,281][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:26:53,606][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:26:53,937][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:26:54,257][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:26:54,583][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:26:54,909][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:26:55,235][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:26:55,562][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:26:55,888][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:26:56,214][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:26:56,545][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:26:56,866][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:26:57,617][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:26:58,396][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:26:58,397][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:26:58,399][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:26:59,430][__main__][INFO] - Iteration 65 took 19s (27.83% Gen, 66.98% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 12m 3s. Estimated total time: 16h 35m 14s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 10s, 500 more iterations: 2h 45m 52s.
[2025-11-13 08:26:59,432][__main__][INFO] - Starting iteration 65.
[2025-11-13 08:26:59,436][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:26:59,437][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:04,933][__main__][INFO] - Number of regex retries in iteration 65: 0
[2025-11-13 08:27:04,933][__main__][INFO] - agents played in iteration 65 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:27:05,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,443][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:05,524][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:05,525][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:06,325][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:27:06,623][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:27:06,950][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:27:07,280][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:27:07,605][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:27:07,931][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:27:08,257][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:27:08,595][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:27:08,924][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:27:09,249][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:27:09,576][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:27:09,908][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:27:10,234][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:27:10,560][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:27:10,886][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:27:11,222][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:27:11,548][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:27:11,876][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:27:12,202][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:27:12,528][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:27:12,855][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:27:13,183][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:27:13,508][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:27:13,835][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:27:14,161][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:27:14,488][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:27:14,816][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:27:15,148][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:27:15,478][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:27:15,807][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:27:16,140][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:27:16,469][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:27:16,795][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:27:17,562][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:27:18,324][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:18,326][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:18,327][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:19,343][__main__][INFO] - Iteration 66 took 19s (27.61% Gen, 67.28% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 11m 53s. Estimated total time: 16h 35m 24s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 10s, 500 more iterations: 2h 45m 54s.
[2025-11-13 08:27:19,346][__main__][INFO] - Starting iteration 66.
[2025-11-13 08:27:19,349][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:27:19,349][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:24,906][__main__][INFO] - Number of regex retries in iteration 66: 0
[2025-11-13 08:27:24,907][__main__][INFO] - agents played in iteration 66 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:27:25,374][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:25,497][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:25,498][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:26,298][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:27:26,596][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:27:26,922][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:27:27,249][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:27:27,578][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:27:27,903][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:27:28,229][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:27:28,557][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:27:28,883][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:27:29,211][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:27:29,542][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:27:29,866][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:27:30,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:27:30,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:27:30,855][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:27:31,178][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:27:31,504][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:27:31,830][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:27:32,159][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:27:32,483][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:27:32,813][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:27:33,139][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:27:33,468][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:27:33,794][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:27:34,123][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:27:34,449][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:27:34,775][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:27:35,102][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:27:35,428][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:27:35,754][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:27:36,080][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:27:36,411][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:27:36,738][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:27:37,509][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:27:38,287][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:38,288][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:38,290][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:39,360][__main__][INFO] - Iteration 67 took 20s (27.77% Gen, 66.87% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 16m 45s. Estimated total time: 16h 40m 36s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 21s, 500 more iterations: 2h 46m 46s.
[2025-11-13 08:27:39,362][__main__][INFO] - Starting iteration 67.
[2025-11-13 08:27:39,366][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:27:39,366][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:27:44,907][__main__][INFO] - Number of regex retries in iteration 67: 0
[2025-11-13 08:27:44,908][__main__][INFO] - agents played in iteration 67 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:27:45,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:27:45,494][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:27:45,494][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:27:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:27:46,567][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:27:46,895][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:27:47,223][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:27:47,549][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:27:47,874][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:27:48,202][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:27:48,529][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:27:48,858][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:27:49,184][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:27:49,514][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:27:49,839][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:27:50,164][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:27:50,491][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:27:50,822][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:27:51,148][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:27:51,475][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:27:51,804][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:27:52,144][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:27:52,473][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:27:52,800][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:27:53,126][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:27:53,456][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:27:53,783][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:27:54,109][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:27:54,439][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:27:54,763][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:27:55,089][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:27:55,418][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:27:55,750][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:27:56,072][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:27:56,397][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:27:56,722][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:27:57,479][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:27:58,231][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:27:58,232][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:27:58,234][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:27:59,273][__main__][INFO] - Iteration 68 took 19s (27.83% Gen, 66.94% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 11m 13s. Estimated total time: 16h 35m 23s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 10s, 500 more iterations: 2h 45m 53s.
[2025-11-13 08:27:59,275][__main__][INFO] - Starting iteration 68.
[2025-11-13 08:27:59,278][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:27:59,279][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:04,913][__main__][INFO] - Number of regex retries in iteration 68: 0
[2025-11-13 08:28:04,913][__main__][INFO] - agents played in iteration 68 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:28:05,395][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:05,515][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:05,516][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:06,281][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:28:06,578][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:28:06,903][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:28:07,228][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:28:07,553][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:28:07,879][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:28:08,210][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:28:08,535][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:28:08,861][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:28:09,186][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:28:09,512][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:28:09,840][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:28:10,167][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:28:10,492][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:28:10,821][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:28:11,148][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:28:11,475][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:28:11,801][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:28:12,127][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:28:12,453][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:28:12,781][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:28:13,111][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:28:13,437][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:28:13,764][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:28:14,090][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:28:14,415][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:28:14,742][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:28:15,068][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:28:15,394][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:28:15,724][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:28:16,051][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:28:16,378][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:28:16,704][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:28:17,479][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:28:18,238][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:18,240][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:18,241][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:28:19,275][__main__][INFO] - Iteration 69 took 19s (28.17% Gen, 66.65% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 15m 21s. Estimated total time: 16h 39m 52s. Time estimates for 10 more iterations: 3m 19s, 100 more iterations: 33m 19s, 500 more iterations: 2h 46m 38s.
[2025-11-13 08:28:19,277][__main__][INFO] - Starting iteration 69.
[2025-11-13 08:28:19,281][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:28:19,281][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:24,827][__main__][INFO] - Number of regex retries in iteration 69: 0
[2025-11-13 08:28:24,828][__main__][INFO] - agents played in iteration 69 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:28:25,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:25,413][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:25,414][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:26,187][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:28:26,484][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:28:26,810][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:28:27,139][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:28:27,472][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:28:27,801][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:28:28,129][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:28:28,455][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:28:28,785][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:28:29,114][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:28:29,443][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:28:29,774][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:28:30,103][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:28:30,431][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:28:30,764][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:28:31,090][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:28:31,420][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:28:31,749][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:28:32,083][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:28:32,415][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:28:32,745][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:28:33,071][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:28:33,398][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:28:33,724][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:28:34,051][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:28:34,378][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:28:34,704][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:28:35,030][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:28:35,361][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:28:35,683][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:28:36,010][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:28:36,336][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:28:36,663][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:28:37,438][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:28:38,228][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:38,229][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:38,231][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:28:39,280][__main__][INFO] - Iteration 70 took 20s (27.73% Gen, 67.02% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 15m 10s. Estimated total time: 16h 40m 1s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 20s, 500 more iterations: 2h 46m 40s.
[2025-11-13 08:28:39,282][__main__][INFO] - Starting iteration 70.
[2025-11-13 08:28:39,285][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 6 and human policies 1.
[2025-11-13 08:28:39,286][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:28:44,745][__main__][INFO] - Number of regex retries in iteration 70: 0
[2025-11-13 08:28:44,746][__main__][INFO] - agents played in iteration 70 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:28:45,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:28:45,336][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:28:45,337][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:28:46,132][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:28:46,429][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:28:46,758][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:28:47,091][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:28:47,422][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:28:47,748][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:28:48,080][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:28:48,414][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:28:48,751][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:28:49,080][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:28:49,410][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:28:49,737][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:28:50,063][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:28:50,397][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:28:50,726][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:28:51,060][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:28:51,389][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:28:51,718][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:28:52,046][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:28:52,372][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:28:52,698][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:28:53,028][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:28:53,357][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:28:53,686][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:28:54,022][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:28:54,348][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:28:54,674][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:28:55,002][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:28:55,332][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:28:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:28:55,984][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:28:56,310][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:28:56,639][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:28:57,394][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:28:58,161][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:28:58,163][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:28:58,164][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:29:00,180][__main__][INFO] - Iteration 71 took 20s (26.12% Gen, 64.22% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 59m 36s. Estimated total time: 17h 24m 47s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 49s, 500 more iterations: 2h 54m 7s.
[2025-11-13 08:29:00,183][__main__][INFO] - Starting iteration 71.
[2025-11-13 08:29:00,185][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:29:00,186][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:06,245][__main__][INFO] - Number of regex retries in iteration 71: 0
[2025-11-13 08:29:06,245][__main__][INFO] - agents played in iteration 71 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:29:06,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,752][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,831][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:06,832][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:06,832][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:29:07,614][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:29:07,913][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:29:08,240][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:29:08,569][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:29:08,895][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:29:09,222][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:29:09,551][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:29:09,877][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:29:10,203][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:29:10,533][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:29:10,858][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:29:11,187][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:29:11,516][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:29:11,842][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:29:12,168][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:29:12,494][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:29:12,820][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:29:13,146][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:29:13,474][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:29:13,802][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:29:14,130][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:29:14,456][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:29:14,783][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:29:15,108][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:29:15,435][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:29:15,762][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:29:16,091][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:29:16,422][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:29:16,746][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:29:17,072][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:29:17,400][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:29:17,737][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:29:18,064][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:29:18,809][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:29:19,561][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:29:19,563][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:29:19,564][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:29:20,657][__main__][INFO] - Iteration 72 took 20s (29.59% Gen, 65.06% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 38m 4s. Estimated total time: 17h 3m 36s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 7s, 500 more iterations: 2h 50m 36s.
[2025-11-13 08:29:20,659][__main__][INFO] - Starting iteration 72.
[2025-11-13 08:29:20,662][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:29:20,662][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:26,452][__main__][INFO] - Number of regex retries in iteration 72: 0
[2025-11-13 08:29:26,453][__main__][INFO] - agents played in iteration 72 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:29:26,931][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:26,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:27,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:27,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:27,052][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:27,052][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:29:27,822][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:29:28,125][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:29:28,457][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:29:28,782][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:29:29,109][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:29:29,444][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:29:29,770][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:29:30,099][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:29:30,425][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:29:30,761][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:29:31,086][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:29:31,412][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:29:31,738][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:29:32,064][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:29:32,391][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:29:32,717][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:29:33,043][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:29:33,372][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:29:33,698][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:29:34,024][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:29:34,351][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:29:34,677][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:29:35,002][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:29:35,329][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:29:35,656][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:29:35,985][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:29:36,314][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:29:36,642][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:29:36,967][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:29:37,293][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:29:37,620][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:29:37,948][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:29:38,275][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:29:39,040][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:29:39,815][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:29:39,817][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:29:39,818][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:29:40,858][__main__][INFO] - Iteration 73 took 20s (28.67% Gen, 66.18% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 23m 59s. Estimated total time: 16h 49m 51s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 39s, 500 more iterations: 2h 48m 18s.
[2025-11-13 08:29:40,860][__main__][INFO] - Starting iteration 73.
[2025-11-13 08:29:40,862][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:29:40,863][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:29:46,647][__main__][INFO] - Number of regex retries in iteration 73: 0
[2025-11-13 08:29:46,648][__main__][INFO] - agents played in iteration 73 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:29:47,124][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,165][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,245][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:29:47,245][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:29:47,246][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:29:48,029][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:29:48,327][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:29:48,654][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:29:48,983][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:29:49,316][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:29:49,649][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:29:49,978][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:29:50,307][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:29:50,633][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:29:50,961][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:29:51,289][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:29:51,615][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:29:51,941][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:29:52,267][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:29:52,594][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:29:52,921][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:29:53,247][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:29:53,573][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:29:53,900][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:29:54,226][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:29:54,553][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:29:54,880][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:29:55,206][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:29:55,538][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:29:55,867][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:29:56,200][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:29:56,529][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:29:56,859][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:29:57,184][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:29:57,509][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:29:57,836][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:29:58,166][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:29:58,500][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:29:59,261][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:30:00,028][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:00,029][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:00,031][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:01,217][__main__][INFO] - Iteration 74 took 20s (28.42% Gen, 65.75% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 31m 34s. Estimated total time: 16h 57m 46s. Time estimates for 10 more iterations: 3m 23s, 100 more iterations: 33m 55s, 500 more iterations: 2h 49m 37s.
[2025-11-13 08:30:01,219][__main__][INFO] - Starting iteration 74.
[2025-11-13 08:30:01,222][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:30:01,223][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:30:06,906][__main__][INFO] - Number of regex retries in iteration 74: 0
[2025-11-13 08:30:06,907][__main__][INFO] - agents played in iteration 74 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:30:07,383][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:07,505][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:30:07,506][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:08,291][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:30:08,589][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:30:08,916][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:30:09,244][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:30:09,572][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:30:09,897][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:30:10,223][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:30:10,550][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:30:10,879][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:30:11,206][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:30:11,533][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:30:11,859][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:30:12,185][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:30:12,512][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:30:12,839][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:30:13,168][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:30:13,495][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:30:13,821][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:30:14,148][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:30:14,473][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:30:14,803][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:30:15,132][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:30:15,463][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:30:15,793][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:30:16,120][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:30:16,446][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:30:16,776][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:30:17,104][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:30:17,432][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:30:17,759][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:30:18,084][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:30:18,412][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:30:18,738][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:30:19,503][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:30:20,262][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:20,263][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:20,265][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:21,287][__main__][INFO] - Iteration 75 took 20s (28.33% Gen, 66.57% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 16m 45s. Estimated total time: 16h 43m 17s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 26s, 500 more iterations: 2h 47m 12s.
[2025-11-13 08:30:21,289][__main__][INFO] - Starting iteration 75.
[2025-11-13 08:30:21,293][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:30:21,294][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:30:27,133][__main__][INFO] - Number of regex retries in iteration 75: 0
[2025-11-13 08:30:27,133][__main__][INFO] - agents played in iteration 75 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:30:27,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:27,719][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:30:27,719][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:28,503][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:30:28,800][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:30:29,129][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:30:29,460][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:30:29,781][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:30:30,107][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:30:30,433][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:30:30,763][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:30:31,087][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:30:31,412][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:30:31,738][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:30:32,066][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:30:32,393][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:30:32,721][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:30:33,050][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:30:33,377][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:30:33,710][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:30:34,037][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:30:34,363][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:30:34,689][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:30:35,016][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:30:35,341][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:30:35,668][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:30:35,995][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:30:36,323][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:30:36,650][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:30:36,977][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:30:37,303][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:30:37,629][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:30:37,956][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:30:38,283][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:30:38,609][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:30:38,941][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:30:39,705][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:30:40,467][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:30:40,469][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:30:40,471][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:30:41,468][__main__][INFO] - Iteration 76 took 20s (28.94% Gen, 66.11% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 21m 54s. Estimated total time: 16h 48m 46s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 37s, 500 more iterations: 2h 48m 7s.
[2025-11-13 08:30:41,470][__main__][INFO] - Starting iteration 76.
[2025-11-13 08:30:41,474][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:30:41,474][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:30:47,233][__main__][INFO] - Number of regex retries in iteration 76: 0
[2025-11-13 08:30:47,233][__main__][INFO] - agents played in iteration 76 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:30:47,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:47,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:47,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:47,840][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:30:47,840][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:30:47,841][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:30:48,606][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:30:48,905][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:30:49,230][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:30:49,556][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:30:49,883][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:30:50,208][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:30:50,534][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:30:50,861][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:30:51,192][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:30:51,516][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:30:51,842][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:30:52,177][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:30:52,503][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:30:52,830][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:30:53,157][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:30:53,491][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:30:53,820][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:30:54,146][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:30:54,480][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:30:54,805][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:30:55,134][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:30:55,460][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:30:55,790][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:30:56,119][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:30:56,444][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:30:56,771][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:30:57,097][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:30:57,423][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:30:57,749][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:30:58,079][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:30:58,407][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:30:58,741][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:30:59,074][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:30:59,836][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:31:00,581][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:00,583][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:00,584][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:01,584][__main__][INFO] - Iteration 77 took 20s (28.63% Gen, 66.39% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 18m 21s. Estimated total time: 16h 45m 33s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 31s, 500 more iterations: 2h 47m 35s.
[2025-11-13 08:31:01,586][__main__][INFO] - Starting iteration 77.
[2025-11-13 08:31:01,590][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:31:01,591][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:07,498][__main__][INFO] - Number of regex retries in iteration 77: 0
[2025-11-13 08:31:07,498][__main__][INFO] - agents played in iteration 77 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:31:07,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:08,103][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:08,104][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:08,881][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:31:09,180][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:31:09,507][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:31:09,831][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:31:10,160][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:31:10,487][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:31:10,813][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:31:11,139][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:31:11,465][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:31:11,791][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:31:12,116][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:31:12,442][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:31:12,768][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:31:13,095][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:31:13,422][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:31:13,748][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:31:14,076][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:31:14,404][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:31:14,730][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:31:15,057][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:31:15,384][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:31:15,710][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:31:16,036][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:31:16,362][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:31:16,692][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:31:17,020][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:31:17,346][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:31:17,673][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:31:18,000][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:31:18,326][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:31:18,652][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:31:18,978][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:31:19,305][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:31:20,065][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:31:20,818][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:20,820][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:20,822][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:21,827][__main__][INFO] - Iteration 78 took 20s (29.19% Gen, 65.84% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 24m 21s. Estimated total time: 16h 51m 54s. Time estimates for 10 more iterations: 3m 22s, 100 more iterations: 33m 43s, 500 more iterations: 2h 48m 39s.
[2025-11-13 08:31:21,829][__main__][INFO] - Starting iteration 78.
[2025-11-13 08:31:21,833][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:31:21,833][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:27,626][__main__][INFO] - Number of regex retries in iteration 78: 0
[2025-11-13 08:31:27,627][__main__][INFO] - agents played in iteration 78 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:31:28,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,162][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:28,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:28,203][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:28,978][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:31:29,277][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:31:29,605][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:31:29,933][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:31:30,264][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:31:30,591][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:31:30,917][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:31:31,245][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:31:31,573][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:31:31,902][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:31:32,229][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:31:32,555][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:31:32,887][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:31:33,210][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:31:33,536][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:31:33,861][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:31:34,189][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:31:34,522][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:31:34,852][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:31:35,177][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:31:35,504][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:31:35,833][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:31:36,162][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:31:36,495][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:31:36,822][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:31:37,150][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:31:37,477][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:31:37,805][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:31:38,131][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:31:38,458][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:31:38,784][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:31:39,113][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:31:39,439][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:31:40,196][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:31:40,983][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:31:40,985][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:31:40,986][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:31:41,996][__main__][INFO] - Iteration 79 took 20s (28.73% Gen, 66.26% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 20m 18s. Estimated total time: 16h 48m 11s. Time estimates for 10 more iterations: 3m 21s, 100 more iterations: 33m 36s, 500 more iterations: 2h 48m 1s.
[2025-11-13 08:31:41,998][__main__][INFO] - Starting iteration 79.
[2025-11-13 08:31:42,002][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:31:42,003][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:31:47,730][__main__][INFO] - Number of regex retries in iteration 79: 0
[2025-11-13 08:31:47,731][__main__][INFO] - agents played in iteration 79 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:31:48,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,231][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:31:48,312][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:31:48,313][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:31:49,091][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:31:49,390][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:31:49,717][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:31:50,041][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:31:50,369][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:31:50,695][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:31:51,027][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:31:51,361][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:31:51,686][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:31:52,013][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:31:52,338][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:31:52,671][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:31:52,997][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:31:53,325][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:31:53,651][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:31:53,991][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:31:54,320][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:31:54,647][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:31:54,972][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:31:55,302][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:31:55,628][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:31:55,956][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:31:56,284][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:31:56,612][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:31:56,939][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:31:57,264][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:31:57,598][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:31:57,935][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:31:58,261][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:31:58,587][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:31:58,913][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:31:59,238][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:31:59,564][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:32:00,323][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:32:01,063][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:01,065][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:01,066][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:02,089][__main__][INFO] - Iteration 80 took 20s (28.51% Gen, 66.39% Train). Generation: 5s, Training: 13s. Estimated remaining time: 16h 16m 9s. Estimated total time: 16h 44m 22s. Time estimates for 10 more iterations: 3m 20s, 100 more iterations: 33m 28s, 500 more iterations: 2h 47m 23s.
[2025-11-13 08:32:02,091][__main__][INFO] - Starting iteration 80.
[2025-11-13 08:32:02,095][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 7 and human policies 1.
[2025-11-13 08:32:02,095][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:07,862][__main__][INFO] - Number of regex retries in iteration 80: 0
[2025-11-13 08:32:07,863][__main__][INFO] - agents played in iteration 80 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:32:08,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:08,446][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:08,447][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:09,216][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:32:09,519][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:32:09,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:32:10,171][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:32:10,498][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:32:10,826][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:32:11,155][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:32:11,482][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:32:11,809][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:32:12,135][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:32:12,464][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:32:12,791][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:32:13,123][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:32:13,457][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:32:13,794][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:32:14,124][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:32:14,450][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:32:14,776][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:32:15,109][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:32:15,439][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:32:15,767][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:32:16,099][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:32:16,420][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:32:16,745][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:32:17,072][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:32:17,404][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:32:17,725][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:32:18,050][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:32:18,376][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:32:18,702][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:32:19,028][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:32:19,354][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:32:19,680][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:32:20,435][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:32:21,196][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:21,197][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:21,199][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:23,206][__main__][INFO] - Iteration 81 took 21s (27.32% Gen, 63.17% Train). Generation: 5s, Training: 13s. Estimated remaining time: 17h 7m 2s. Estimated total time: 17h 35m 37s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 11s, 500 more iterations: 2h 55m 56s.
[2025-11-13 08:32:23,208][__main__][INFO] - Starting iteration 81.
[2025-11-13 08:32:23,212][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:32:23,212][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:29,592][__main__][INFO] - Number of regex retries in iteration 81: 0
[2025-11-13 08:32:29,593][__main__][INFO] - agents played in iteration 81 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:32:30,056][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:30,178][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:30,178][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:30,962][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:32:31,261][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:32:31,588][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:32:31,917][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:32:32,243][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:32:32,569][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:32:32,896][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:32:33,224][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:32:33,550][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:32:33,877][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:32:34,207][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:32:34,536][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:32:34,864][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:32:35,190][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:32:35,524][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:32:35,854][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:32:36,180][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:32:36,506][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:32:36,832][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:32:37,158][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:32:37,494][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:32:37,821][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:32:38,149][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:32:38,476][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:32:38,805][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:32:39,130][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:32:39,457][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:32:39,787][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:32:40,116][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:32:40,443][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:32:40,776][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:32:41,106][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:32:41,427][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:32:42,182][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:32:42,949][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:32:42,951][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:32:42,953][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:32:43,975][__main__][INFO] - Iteration 82 took 20s (30.73% Gen, 64.34% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 49m 17s. Estimated total time: 17h 18m 12s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 36s, 500 more iterations: 2h 53m 2s.
[2025-11-13 08:32:43,978][__main__][INFO] - Starting iteration 82.
[2025-11-13 08:32:43,981][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:32:43,981][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:32:49,996][__main__][INFO] - Number of regex retries in iteration 82: 0
[2025-11-13 08:32:49,997][__main__][INFO] - agents played in iteration 82 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:32:50,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:32:50,590][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:32:50,591][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:32:51,363][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:32:51,660][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:32:51,987][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:32:52,314][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:32:52,640][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:32:52,971][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:32:53,300][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:32:53,627][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:32:53,954][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:32:54,279][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:32:54,604][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:32:54,932][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:32:55,258][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:32:55,586][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:32:55,914][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:32:56,240][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:32:56,565][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:32:56,893][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:32:57,222][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:32:57,551][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:32:57,876][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:32:58,206][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:32:58,528][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:32:58,857][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:32:59,183][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:32:59,514][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:32:59,840][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:33:00,165][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:33:00,495][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:33:00,820][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:33:01,147][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:33:01,475][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:33:01,803][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:33:02,576][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:33:03,336][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:03,338][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:03,340][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:04,373][__main__][INFO] - Iteration 83 took 20s (29.50% Gen, 65.43% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 30m 22s. Estimated total time: 16h 59m 38s. Time estimates for 10 more iterations: 3m 23s, 100 more iterations: 33m 59s, 500 more iterations: 2h 49m 56s.
[2025-11-13 08:33:04,375][__main__][INFO] - Starting iteration 83.
[2025-11-13 08:33:04,380][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:33:04,380][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:10,563][__main__][INFO] - Number of regex retries in iteration 83: 0
[2025-11-13 08:33:10,564][__main__][INFO] - agents played in iteration 83 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:33:11,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,126][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,167][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:11,167][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:11,167][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:11,939][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:33:12,237][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:33:12,564][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:33:12,901][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:33:13,229][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:33:13,555][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:33:13,883][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:33:14,212][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:33:14,542][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:33:14,867][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:33:15,199][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:33:15,526][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:33:15,855][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:33:16,184][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:33:16,515][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:33:16,836][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:33:17,161][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:33:17,487][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:33:17,818][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:33:18,139][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:33:18,465][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:33:18,791][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:33:19,122][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:33:19,442][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:33:19,769][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:33:20,095][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:33:20,424][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:33:20,754][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:33:21,086][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:33:21,412][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:33:21,738][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:33:22,064][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:33:22,391][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:33:23,152][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:33:23,916][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:23,918][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:23,920][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:24,934][__main__][INFO] - Iteration 84 took 20s (30.08% Gen, 64.98% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 38m 11s. Estimated total time: 17h 7m 47s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 15s, 500 more iterations: 2h 51m 17s.
[2025-11-13 08:33:24,937][__main__][INFO] - Starting iteration 84.
[2025-11-13 08:33:24,940][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:33:24,940][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:30,973][__main__][INFO] - Number of regex retries in iteration 84: 0
[2025-11-13 08:33:30,974][__main__][INFO] - agents played in iteration 84 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:33:31,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:31,558][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:31,558][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:32,327][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:33:32,625][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:33:32,956][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:33:33,282][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:33:33,611][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:33:33,939][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:33:34,266][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:33:34,591][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:33:34,922][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:33:35,244][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:33:35,570][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:33:35,896][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:33:36,227][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:33:36,553][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:33:36,887][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:33:37,216][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:33:37,545][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:33:37,870][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:33:38,197][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:33:38,523][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:33:38,849][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:33:39,177][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:33:39,504][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:33:39,830][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:33:40,156][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:33:40,483][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:33:40,810][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:33:41,136][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:33:41,464][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:33:41,790][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:33:42,117][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:33:42,446][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:33:42,773][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:33:43,548][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:33:44,335][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:33:44,337][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:33:44,339][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:33:45,413][__main__][INFO] - Iteration 85 took 20s (29.47% Gen, 65.28% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 33m 46s. Estimated total time: 17h 3m 42s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 7s, 500 more iterations: 2h 50m 37s.
[2025-11-13 08:33:45,416][__main__][INFO] - Starting iteration 85.
[2025-11-13 08:33:45,419][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:33:45,420][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:33:51,467][__main__][INFO] - Number of regex retries in iteration 85: 0
[2025-11-13 08:33:51,468][__main__][INFO] - agents played in iteration 85 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:33:51,932][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:51,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:33:52,056][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:33:52,056][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:33:52,845][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:33:53,142][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:33:53,468][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:33:53,795][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:33:54,127][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:33:54,457][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:33:54,790][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:33:55,117][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:33:55,443][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:33:55,771][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:33:56,098][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:33:56,425][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:33:56,752][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:33:57,078][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:33:57,405][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:33:57,732][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:33:58,058][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:33:58,385][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:33:58,712][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:33:59,039][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:33:59,365][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:33:59,692][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:34:00,018][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:34:00,345][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:34:00,671][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:34:00,999][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:34:01,324][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:34:01,651][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:34:01,978][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:34:02,305][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:34:02,633][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:34:02,961][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:34:03,287][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:34:04,063][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:34:04,842][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:04,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:04,846][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:05,869][__main__][INFO] - Iteration 86 took 20s (29.57% Gen, 65.42% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 32m 15s. Estimated total time: 17h 2m 32s. Time estimates for 10 more iterations: 3m 24s, 100 more iterations: 34m 5s, 500 more iterations: 2h 50m 25s.
[2025-11-13 08:34:05,871][__main__][INFO] - Starting iteration 86.
[2025-11-13 08:34:05,875][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:34:05,875][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:11,988][__main__][INFO] - Number of regex retries in iteration 86: 0
[2025-11-13 08:34:11,988][__main__][INFO] - agents played in iteration 86 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:34:12,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:12,575][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:12,575][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:13,352][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:34:13,649][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:34:13,977][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:34:14,303][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:34:14,635][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:34:14,964][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:34:15,290][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:34:15,626][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:34:15,950][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:34:16,279][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:34:16,605][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:34:16,934][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:34:17,263][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:34:17,595][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:34:17,925][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:34:18,251][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:34:18,581][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:34:18,910][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:34:19,239][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:34:19,563][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:34:19,889][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:34:20,215][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:34:20,542][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:34:20,868][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:34:21,195][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:34:21,525][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:34:21,855][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:34:22,181][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:34:22,510][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:34:22,837][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:34:23,163][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:34:23,489][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:34:23,817][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:34:24,594][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:34:25,354][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:25,355][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:25,357][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:26,606][__main__][INFO] - Iteration 87 took 20s (29.49% Gen, 64.48% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 45m 59s. Estimated total time: 17h 16m 36s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 33s, 500 more iterations: 2h 52m 46s.
[2025-11-13 08:34:26,608][__main__][INFO] - Starting iteration 87.
[2025-11-13 08:34:26,611][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:34:26,611][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:32,720][__main__][INFO] - Number of regex retries in iteration 87: 0
[2025-11-13 08:34:32,721][__main__][INFO] - agents played in iteration 87 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:34:33,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:33,308][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:33,308][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:34,100][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:34:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:34:34,725][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:34:35,054][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:34:35,380][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:34:35,706][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:34:36,033][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:34:36,366][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:34:36,695][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:34:37,024][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:34:37,351][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:34:37,678][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:34:38,017][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:34:38,344][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:34:38,671][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:34:39,001][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:34:39,332][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:34:39,663][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:34:39,991][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:34:40,323][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:34:40,646][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:34:40,975][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:34:41,304][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:34:41,635][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:34:41,957][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:34:42,285][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:34:42,611][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:34:42,939][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:34:43,265][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:34:43,592][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:34:43,918][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:34:44,251][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:34:44,592][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:34:45,333][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:34:46,113][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:34:46,114][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:34:46,116][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:34:47,132][__main__][INFO] - Iteration 88 took 20s (29.77% Gen, 65.27% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 35m 6s. Estimated total time: 17h 6m 4s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 12s, 500 more iterations: 2h 51m 0s.
[2025-11-13 08:34:47,134][__main__][INFO] - Starting iteration 88.
[2025-11-13 08:34:47,137][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:34:47,137][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:34:53,157][__main__][INFO] - Number of regex retries in iteration 88: 0
[2025-11-13 08:34:53,158][__main__][INFO] - agents played in iteration 88 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:34:53,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:34:53,747][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:34:53,747][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:34:54,494][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:34:54,793][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:34:55,130][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:34:55,457][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:34:55,783][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:34:56,112][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:34:56,441][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:34:56,776][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:34:57,110][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:34:57,443][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:34:57,768][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:34:58,098][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:34:58,426][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:34:58,754][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:34:59,080][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:34:59,409][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:34:59,736][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:35:00,067][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:35:00,394][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:35:00,721][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:35:01,048][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:35:01,376][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:35:01,704][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:35:02,030][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:35:02,359][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:35:02,687][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:35:03,015][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:35:03,345][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:35:03,671][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:35:03,998][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:35:04,328][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:35:04,656][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:35:04,986][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:35:05,732][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:35:06,476][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:35:06,478][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:35:06,480][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:35:07,689][__main__][INFO] - Iteration 89 took 20s (29.29% Gen, 64.82% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 36m 21s. Estimated total time: 17h 7m 40s. Time estimates for 10 more iterations: 3m 25s, 100 more iterations: 34m 15s, 500 more iterations: 2h 51m 16s.
[2025-11-13 08:35:07,691][__main__][INFO] - Starting iteration 89.
[2025-11-13 08:35:07,695][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:35:07,696][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:13,848][__main__][INFO] - Number of regex retries in iteration 89: 0
[2025-11-13 08:35:13,849][__main__][INFO] - agents played in iteration 89 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:35:14,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:14,434][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:14,434][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:15,216][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:35:15,515][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:35:15,845][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:35:16,169][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:35:16,496][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:35:16,823][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:35:17,150][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:35:17,479][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:35:17,806][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:35:18,133][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:35:18,461][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:35:18,791][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:35:19,117][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:35:19,444][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:35:19,770][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:35:20,109][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:35:20,438][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:35:20,769][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:35:21,103][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:35:21,432][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:35:21,761][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:35:22,089][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:35:22,419][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:35:22,748][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:35:23,077][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:35:23,407][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:35:23,736][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:35:24,066][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:35:24,399][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:35:24,729][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:35:25,056][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:35:25,386][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:35:25,712][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:35:26,483][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:35:27,257][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:35:27,258][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:35:27,260][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:35:28,395][__main__][INFO] - Iteration 90 took 20s (29.73% Gen, 64.79% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 43m 22s. Estimated total time: 17h 15m 1s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 30s, 500 more iterations: 2h 52m 30s.
[2025-11-13 08:35:28,397][__main__][INFO] - Starting iteration 90.
[2025-11-13 08:35:28,400][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 8 and human policies 1.
[2025-11-13 08:35:28,400][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:34,426][__main__][INFO] - Number of regex retries in iteration 90: 0
[2025-11-13 08:35:34,427][__main__][INFO] - agents played in iteration 90 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:35:34,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:34,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:34,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:35,014][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:35,015][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:35,015][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:35,812][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:35:36,110][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:35:36,442][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:35:36,775][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:35:37,109][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:35:37,442][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:35:37,769][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:35:38,096][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:35:38,423][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:35:38,753][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:35:39,080][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:35:39,414][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:35:39,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:35:40,067][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:35:40,395][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:35:40,730][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:35:41,054][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:35:41,380][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:35:41,708][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:35:42,041][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:35:42,367][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:35:42,696][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:35:43,023][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:35:43,353][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:35:43,680][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:35:44,008][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:35:44,337][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:35:44,665][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:35:45,004][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:35:45,331][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:35:45,657][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:35:45,983][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:35:46,327][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:35:47,083][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:35:47,863][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:35:47,864][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:35:47,866][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:35:49,961][__main__][INFO] - Iteration 91 took 21s (27.95% Gen, 62.33% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 26m 4s. Estimated total time: 17h 58m 5s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 56s, 500 more iterations: 2h 59m 40s.
[2025-11-13 08:35:49,963][__main__][INFO] - Starting iteration 91.
[2025-11-13 08:35:49,966][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:35:49,966][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:35:56,488][__main__][INFO] - Number of regex retries in iteration 91: 0
[2025-11-13 08:35:56,489][__main__][INFO] - agents played in iteration 91 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:35:56,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:57,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:57,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:57,082][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:35:57,082][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:35:57,083][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:35:57,830][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:35:58,128][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:35:58,460][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:35:58,788][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:35:59,121][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:35:59,450][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:35:59,778][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:36:00,104][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:36:00,434][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:36:00,759][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:36:01,087][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:36:01,414][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:36:01,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:36:02,068][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:36:02,395][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:36:02,722][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:36:03,048][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:36:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:36:03,701][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:36:04,027][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:36:04,354][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:36:04,681][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:36:05,010][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:36:05,337][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:36:05,664][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:36:05,990][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:36:06,332][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:36:06,658][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:36:06,986][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:36:07,313][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:36:07,645][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:36:07,971][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:36:08,300][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:36:09,062][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:36:09,811][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:09,812][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:09,814][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:10,814][__main__][INFO] - Iteration 92 took 20s (31.28% Gen, 63.91% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 50m 6s. Estimated total time: 17h 22m 28s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 44s, 500 more iterations: 2h 53m 44s.
[2025-11-13 08:36:10,816][__main__][INFO] - Starting iteration 92.
[2025-11-13 08:36:10,819][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:36:10,820][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:17,170][__main__][INFO] - Number of regex retries in iteration 92: 0
[2025-11-13 08:36:17,171][__main__][INFO] - agents played in iteration 92 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:36:17,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:17,760][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:17,760][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:36:18,507][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:36:18,802][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:36:19,131][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:36:19,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:36:19,788][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:36:20,115][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:36:20,445][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:36:20,771][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:36:21,104][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:36:21,431][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:36:21,758][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:36:22,087][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:36:22,413][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:36:22,739][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:36:23,069][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:36:23,396][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:36:23,722][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:36:24,049][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:36:24,375][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:36:24,704][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:36:25,031][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:36:25,358][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:36:25,684][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:36:26,010][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:36:26,338][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:36:26,664][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:36:26,994][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:36:27,321][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:36:27,648][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:36:27,975][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:36:28,302][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:36:28,631][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:36:28,958][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:36:29,693][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:36:30,469][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:30,471][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:30,473][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:31,609][__main__][INFO] - Iteration 93 took 20s (30.55% Gen, 63.98% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 46m 49s. Estimated total time: 17h 19m 32s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 39s, 500 more iterations: 2h 53m 15s.
[2025-11-13 08:36:31,611][__main__][INFO] - Starting iteration 93.
[2025-11-13 08:36:31,614][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:36:31,614][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:37,970][__main__][INFO] - Number of regex retries in iteration 93: 0
[2025-11-13 08:36:37,970][__main__][INFO] - agents played in iteration 93 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:36:38,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:38,557][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:38,558][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:36:39,343][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:36:39,639][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:36:39,966][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:36:40,293][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:36:40,620][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:36:40,947][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:36:41,277][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:36:41,603][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:36:41,930][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:36:42,257][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:36:42,584][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:36:42,912][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:36:43,237][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:36:43,566][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:36:43,892][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:36:44,222][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:36:44,554][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:36:44,884][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:36:45,211][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:36:45,537][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:36:45,863][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:36:46,189][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:36:46,517][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:36:46,842][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:36:47,172][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:36:47,497][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:36:47,826][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:36:48,155][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:36:48,483][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:36:48,813][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:36:49,142][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:36:49,471][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:36:49,800][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:36:50,531][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:36:51,315][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:36:51,317][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:36:51,319][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:36:52,357][__main__][INFO] - Iteration 94 took 20s (30.64% Gen, 64.35% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 44m 9s. Estimated total time: 17h 17m 12s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 34s, 500 more iterations: 2h 52m 52s.
[2025-11-13 08:36:52,359][__main__][INFO] - Starting iteration 94.
[2025-11-13 08:36:52,362][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:36:52,363][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:36:58,764][__main__][INFO] - Number of regex retries in iteration 94: 0
[2025-11-13 08:36:58,765][__main__][INFO] - agents played in iteration 94 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:36:59,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:36:59,354][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:36:59,354][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:00,133][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:37:00,430][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:37:00,762][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:37:01,089][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:37:01,423][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:37:01,750][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:37:02,077][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:37:02,403][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:37:02,742][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:37:03,069][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:37:03,396][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:37:03,725][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:37:04,053][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:37:04,383][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:37:04,712][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:37:05,039][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:37:05,366][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:37:05,692][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:37:06,018][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:37:06,345][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:37:06,670][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:37:06,999][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:37:07,325][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:37:07,656][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:37:07,978][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:37:08,306][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:37:08,634][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:37:08,966][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:37:09,290][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:37:09,617][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:37:09,950][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:37:10,277][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:37:10,608][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:37:11,368][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:37:12,142][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:12,143][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:12,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:13,167][__main__][INFO] - Iteration 95 took 20s (30.77% Gen, 64.31% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 46m 52s. Estimated total time: 17h 20m 17s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 40s, 500 more iterations: 2h 53m 22s.
[2025-11-13 08:37:13,169][__main__][INFO] - Starting iteration 95.
[2025-11-13 08:37:13,173][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:37:13,173][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:37:19,515][__main__][INFO] - Number of regex retries in iteration 95: 0
[2025-11-13 08:37:19,516][__main__][INFO] - agents played in iteration 95 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:37:19,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,017][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:20,098][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:37:20,098][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:20,869][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:37:21,169][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:37:21,496][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:37:21,822][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:37:22,154][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:37:22,481][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:37:22,806][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:37:23,133][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:37:23,460][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:37:23,788][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:37:24,121][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:37:24,453][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:37:24,774][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:37:25,100][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:37:25,427][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:37:25,754][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:37:26,081][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:37:26,410][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:37:26,736][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:37:27,063][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:37:27,389][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:37:27,716][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:37:28,043][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:37:28,370][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:37:28,697][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:37:29,023][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:37:29,349][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:37:29,677][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:37:30,003][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:37:30,330][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:37:30,656][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:37:30,986][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:37:31,313][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:37:32,055][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:37:32,820][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:32,821][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:32,823][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:33,897][__main__][INFO] - Iteration 96 took 20s (30.60% Gen, 64.21% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 42m 28s. Estimated total time: 17h 16m 13s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 32s, 500 more iterations: 2h 52m 42s.
[2025-11-13 08:37:33,899][__main__][INFO] - Starting iteration 96.
[2025-11-13 08:37:33,902][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:37:33,902][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:37:40,201][__main__][INFO] - Number of regex retries in iteration 96: 0
[2025-11-13 08:37:40,201][__main__][INFO] - agents played in iteration 96 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:37:40,661][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:40,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:40,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:40,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:37:40,784][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:37:40,785][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:37:41,569][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:37:41,866][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:37:42,193][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:37:42,519][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:37:42,845][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:37:43,172][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:37:43,500][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:37:43,828][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:37:44,160][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:37:44,487][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:37:44,814][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:37:45,142][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:37:45,469][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:37:45,795][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:37:46,122][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:37:46,450][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:37:46,777][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:37:47,103][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:37:47,430][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:37:47,759][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:37:48,086][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:37:48,414][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:37:48,741][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:37:49,070][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:37:49,396][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:37:49,722][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:37:50,049][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:37:50,379][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:37:50,706][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:37:51,034][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:37:51,360][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:37:51,687][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:37:52,014][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:37:52,746][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:37:53,518][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:37:53,520][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:37:53,521][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:37:54,517][__main__][INFO] - Iteration 97 took 20s (30.55% Gen, 64.61% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 36m 44s. Estimated total time: 17h 10m 50s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 21s, 500 more iterations: 2h 51m 48s.
[2025-11-13 08:37:54,520][__main__][INFO] - Starting iteration 97.
[2025-11-13 08:37:54,523][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:37:54,524][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:00,875][__main__][INFO] - Number of regex retries in iteration 97: 0
[2025-11-13 08:38:00,875][__main__][INFO] - agents played in iteration 97 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:38:01,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:01,473][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:01,473][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:02,242][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:38:02,634][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:38:02,939][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:38:03,265][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:38:03,593][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:38:03,924][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:38:04,251][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:38:04,577][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:38:04,905][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:38:05,232][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:38:05,558][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:38:05,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:38:06,211][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:38:06,538][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:38:06,864][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:38:07,192][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:38:07,523][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:38:07,848][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:38:08,174][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:38:08,500][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:38:08,826][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:38:09,152][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:38:09,482][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:38:09,808][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:38:10,133][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:38:10,459][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:38:10,789][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:38:11,116][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:38:11,446][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:38:11,774][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:38:12,101][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:38:12,429][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:38:12,756][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:38:13,472][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:38:14,228][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:14,230][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:14,232][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:15,232][__main__][INFO] - Iteration 98 took 20s (30.67% Gen, 64.49% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 41m 4s. Estimated total time: 17h 15m 31s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 31s, 500 more iterations: 2h 52m 35s.
[2025-11-13 08:38:15,235][__main__][INFO] - Starting iteration 98.
[2025-11-13 08:38:15,238][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:38:15,239][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:21,668][__main__][INFO] - Number of regex retries in iteration 98: 0
[2025-11-13 08:38:21,668][__main__][INFO] - agents played in iteration 98 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:38:22,148][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,230][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:22,272][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:22,272][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:23,039][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:38:23,338][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:38:23,664][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:38:23,994][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:38:24,325][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:38:24,652][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:38:24,978][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:38:25,307][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:38:25,634][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:38:25,959][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:38:26,286][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:38:26,615][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:38:26,944][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:38:27,274][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:38:27,608][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:38:27,939][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:38:28,264][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:38:28,591][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:38:28,918][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:38:29,249][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:38:29,576][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:38:29,904][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:38:30,231][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:38:30,557][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:38:30,883][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:38:31,210][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:38:31,538][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:38:31,865][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:38:32,192][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:38:32,521][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:38:32,850][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:38:33,177][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:38:33,506][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:38:34,254][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:38:35,019][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:35,021][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:35,022][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:36,289][__main__][INFO] - Iteration 99 took 21s (30.54% Gen, 63.44% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 57m 46s. Estimated total time: 17h 32m 33s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 5s, 500 more iterations: 2h 55m 25s.
[2025-11-13 08:38:36,291][__main__][INFO] - Starting iteration 99.
[2025-11-13 08:38:36,294][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:38:36,295][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:38:42,744][__main__][INFO] - Number of regex retries in iteration 99: 0
[2025-11-13 08:38:42,744][__main__][INFO] - agents played in iteration 99 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:38:43,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,291][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:38:43,332][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:38:43,333][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:38:44,116][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:38:44,414][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:38:44,741][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:38:45,069][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:38:45,396][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:38:45,724][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:38:46,051][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:38:46,376][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:38:46,703][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:38:47,029][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:38:47,355][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:38:47,682][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:38:48,014][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:38:48,344][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:38:48,674][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:38:49,003][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:38:49,330][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:38:49,659][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:38:49,985][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:38:50,315][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:38:50,642][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:38:50,967][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:38:51,294][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:38:51,626][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:38:51,952][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:38:52,280][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:38:52,607][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:38:52,934][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:38:53,260][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:38:53,586][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:38:53,912][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:38:54,238][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:38:54,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:38:55,344][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:38:56,109][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:38:56,111][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:38:56,112][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:38:57,146][__main__][INFO] - Iteration 100 took 20s (30.93% Gen, 64.11% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 47m 29s. Estimated total time: 17h 22m 37s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 45s, 500 more iterations: 2h 53m 46s.
[2025-11-13 08:38:57,148][__main__][INFO] - Starting iteration 100.
[2025-11-13 08:38:57,151][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 9 and human policies 1.
[2025-11-13 08:38:57,152][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:03,542][__main__][INFO] - Number of regex retries in iteration 100: 0
[2025-11-13 08:39:03,542][__main__][INFO] - agents played in iteration 100 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:39:04,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,132][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:04,133][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:04,133][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:04,920][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:39:05,219][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:39:05,557][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:39:05,883][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:39:06,212][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:39:06,539][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:39:06,869][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:39:07,197][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:39:07,524][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:39:07,857][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:39:08,179][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:39:08,506][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:39:08,832][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:39:09,165][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:39:09,487][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:39:09,814][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:39:10,140][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:39:10,479][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:39:10,802][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:39:11,129][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:39:11,462][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:39:11,790][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:39:12,114][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:39:12,440][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:39:12,770][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:39:13,099][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:39:13,427][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:39:13,756][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:39:14,085][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:39:14,415][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:39:14,746][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:39:15,080][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:39:15,406][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:39:16,194][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:39:16,962][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:39:16,964][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:39:16,965][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:39:18,929][__main__][INFO] - Iteration 101 took 21s (29.34% Gen, 61.63% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 33m 26s. Estimated total time: 18h 8m 56s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 17s, 500 more iterations: 3h 1m 29s.
[2025-11-13 08:39:18,931][__main__][INFO] - Starting iteration 101.
[2025-11-13 08:39:18,934][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:39:18,935][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:25,789][__main__][INFO] - Number of regex retries in iteration 101: 0
[2025-11-13 08:39:25,790][__main__][INFO] - agents played in iteration 101 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:39:26,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,383][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:26,383][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:26,384][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:27,143][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:39:27,439][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:39:27,765][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:39:28,091][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:39:28,422][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:39:28,751][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:39:29,083][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:39:29,416][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:39:29,745][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:39:30,073][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:39:30,401][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:39:30,731][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:39:31,064][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:39:31,393][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:39:31,729][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:39:32,058][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:39:32,384][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:39:32,712][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:39:33,047][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:39:33,374][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:39:33,700][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:39:34,028][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:39:34,357][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:39:34,683][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:39:35,008][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:39:35,334][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:39:35,664][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:39:35,990][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:39:36,319][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:39:36,652][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:39:36,986][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:39:37,315][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:39:37,642][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:39:38,384][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:39:39,138][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:39:39,140][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:39:39,142][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:39:40,261][__main__][INFO] - Iteration 102 took 21s (32.14% Gen, 62.60% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 10m 32s. Estimated total time: 17h 46m 23s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 32s, 500 more iterations: 2h 57m 43s.
[2025-11-13 08:39:40,263][__main__][INFO] - Starting iteration 102.
[2025-11-13 08:39:40,266][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:39:40,266][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:39:46,835][__main__][INFO] - Number of regex retries in iteration 102: 0
[2025-11-13 08:39:46,836][__main__][INFO] - agents played in iteration 102 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:39:47,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,447][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:39:47,447][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:39:47,448][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:39:48,233][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:39:48,530][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:39:48,859][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:39:49,186][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:39:49,511][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:39:49,843][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:39:50,173][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:39:50,498][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:39:50,826][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:39:51,159][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:39:51,492][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:39:51,817][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:39:52,142][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:39:52,467][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:39:52,800][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:39:53,128][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:39:53,461][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:39:53,796][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:39:54,126][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:39:54,455][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:39:54,781][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:39:55,107][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:39:55,432][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:39:55,760][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:39:56,093][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:39:56,419][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:39:56,747][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:39:57,076][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:39:57,405][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:39:57,731][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:39:58,067][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:39:58,397][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:39:58,723][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:39:59,451][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:40:00,205][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:00,206][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:00,208][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:01,361][__main__][INFO] - Iteration 103 took 21s (31.14% Gen, 63.39% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 58m 35s. Estimated total time: 17h 34m 48s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 9s, 500 more iterations: 2h 55m 48s.
[2025-11-13 08:40:01,364][__main__][INFO] - Starting iteration 103.
[2025-11-13 08:40:01,366][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:40:01,367][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:40:07,922][__main__][INFO] - Number of regex retries in iteration 103: 0
[2025-11-13 08:40:07,922][__main__][INFO] - agents played in iteration 103 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:40:08,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:08,520][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:40:08,520][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:40:09,303][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:40:09,601][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:40:09,927][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:40:10,255][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:40:10,581][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:40:10,907][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:40:11,233][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:40:11,558][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:40:11,885][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:40:12,211][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:40:12,538][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:40:12,863][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:40:13,189][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:40:13,515][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:40:13,841][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:40:14,166][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:40:14,492][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:40:14,818][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:40:15,143][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:40:15,482][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:40:15,810][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:40:16,136][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:40:16,462][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:40:16,791][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:40:17,117][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:40:17,443][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:40:17,768][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:40:18,096][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:40:18,422][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:40:18,748][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:40:19,074][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:40:19,399][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:40:19,725][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:40:20,523][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:40:21,294][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:21,295][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:21,297][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:22,314][__main__][INFO] - Iteration 104 took 20s (31.29% Gen, 63.84% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 50m 53s. Estimated total time: 17h 27m 26s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 54s, 500 more iterations: 2h 54m 34s.
[2025-11-13 08:40:22,317][__main__][INFO] - Starting iteration 104.
[2025-11-13 08:40:22,320][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:40:22,320][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:40:29,009][__main__][INFO] - Number of regex retries in iteration 104: 0
[2025-11-13 08:40:29,009][__main__][INFO] - agents played in iteration 104 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:40:29,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:29,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:29,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:29,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:29,599][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:40:29,599][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:40:30,380][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:40:30,684][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:40:31,007][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:40:31,342][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:40:31,673][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:40:32,009][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:40:32,333][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:40:32,662][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:40:32,988][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:40:33,314][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:40:33,640][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:40:33,966][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:40:34,291][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:40:34,617][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:40:34,944][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:40:35,271][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:40:35,597][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:40:35,925][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:40:36,255][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:40:36,581][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:40:36,906][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:40:37,233][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:40:37,567][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:40:37,896][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:40:38,221][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:40:38,549][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:40:38,887][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:40:39,212][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:40:39,539][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:40:39,866][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:40:40,192][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:40:40,518][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:40:40,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:40:41,620][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:40:42,374][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:40:42,376][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:40:42,377][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:40:43,517][__main__][INFO] - Iteration 105 took 21s (31.55% Gen, 63.06% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 3m 0s. Estimated total time: 17h 39m 55s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 19s, 500 more iterations: 2h 56m 39s.
[2025-11-13 08:40:43,520][__main__][INFO] - Starting iteration 105.
[2025-11-13 08:40:43,522][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:40:43,523][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:40:50,117][__main__][INFO] - Number of regex retries in iteration 105: 0
[2025-11-13 08:40:50,117][__main__][INFO] - agents played in iteration 105 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:40:50,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:50,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:50,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:50,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:40:50,704][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:40:50,705][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:40:51,483][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:40:51,779][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:40:52,107][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:40:52,434][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:40:52,764][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:40:53,093][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:40:53,423][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:40:53,753][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:40:54,087][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:40:54,422][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:40:54,754][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:40:55,080][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:40:55,412][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:40:55,738][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:40:56,070][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:40:56,401][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:40:56,728][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:40:57,054][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:40:57,379][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:40:57,704][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:40:58,029][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:40:58,355][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:40:58,681][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:40:59,006][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:40:59,332][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:40:59,659][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:40:59,997][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:41:00,323][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:41:00,649][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:41:00,975][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:41:01,305][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:41:01,631][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:41:01,957][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:41:02,690][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:41:03,450][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:03,452][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:03,453][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:04,487][__main__][INFO] - Iteration 106 took 20s (31.45% Gen, 63.61% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 51m 2s. Estimated total time: 17h 28m 18s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 56s, 500 more iterations: 2h 54m 43s.
[2025-11-13 08:41:04,490][__main__][INFO] - Starting iteration 106.
[2025-11-13 08:41:04,494][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:41:04,495][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:11,093][__main__][INFO] - Number of regex retries in iteration 106: 0
[2025-11-13 08:41:11,094][__main__][INFO] - agents played in iteration 106 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:41:11,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:11,601][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:11,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:11,681][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:11,682][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:11,682][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:12,469][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:41:12,767][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:41:13,094][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:41:13,422][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:41:13,751][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:41:14,079][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:41:14,406][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:41:14,732][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:41:15,058][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:41:15,383][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:41:15,716][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:41:16,042][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:41:16,374][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:41:16,700][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:41:17,028][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:41:17,354][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:41:17,685][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:41:18,013][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:41:18,338][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:41:18,666][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:41:18,995][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:41:19,323][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:41:19,647][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:41:19,973][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:41:20,298][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:41:20,626][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:41:20,958][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:41:21,287][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:41:21,612][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:41:21,938][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:41:22,264][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:41:22,590][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:41:22,917][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:41:23,688][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:41:24,436][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:24,437][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:24,439][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:25,464][__main__][INFO] - Iteration 107 took 20s (31.47% Gen, 63.64% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 50m 56s. Estimated total time: 17h 28m 33s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 57s, 500 more iterations: 2h 54m 45s.
[2025-11-13 08:41:25,467][__main__][INFO] - Starting iteration 107.
[2025-11-13 08:41:25,470][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:41:25,470][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:32,068][__main__][INFO] - Number of regex retries in iteration 107: 0
[2025-11-13 08:41:32,069][__main__][INFO] - agents played in iteration 107 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:41:32,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:32,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:32,625][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:32,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:32,668][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:32,668][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:33,457][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:41:33,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:41:34,084][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:41:34,417][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:41:34,745][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:41:35,071][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:41:35,403][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:41:35,733][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:41:36,062][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:41:36,388][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:41:36,714][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:41:37,039][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:41:37,368][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:41:37,693][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:41:38,018][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:41:38,344][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:41:38,670][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:41:38,996][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:41:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:41:39,653][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:41:39,978][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:41:40,304][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:41:40,632][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:41:40,960][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:41:41,288][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:41:41,614][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:41:41,941][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:41:42,265][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:41:42,590][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:41:42,915][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:41:43,240][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:41:43,566][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:41:43,898][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:41:44,692][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:41:45,449][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:41:45,450][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:41:45,452][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:41:46,597][__main__][INFO] - Iteration 108 took 21s (31.23% Gen, 63.35% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 58m 27s. Estimated total time: 17h 36m 25s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 12s, 500 more iterations: 2h 56m 4s.
[2025-11-13 08:41:46,599][__main__][INFO] - Starting iteration 108.
[2025-11-13 08:41:46,603][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:41:46,604][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:41:53,213][__main__][INFO] - Number of regex retries in iteration 108: 0
[2025-11-13 08:41:53,214][__main__][INFO] - agents played in iteration 108 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:41:53,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:53,740][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:53,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:53,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:41:53,821][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:41:53,821][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:41:54,572][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:41:54,870][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:41:55,199][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:41:55,527][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:41:55,854][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:41:56,180][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:41:56,510][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:41:56,836][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:41:57,163][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:41:57,489][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:41:57,821][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:41:58,143][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:41:58,469][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:41:58,795][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:41:59,122][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:41:59,449][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:41:59,774][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:42:00,101][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:42:00,429][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:42:00,761][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:42:01,087][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:42:01,412][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:42:01,738][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:42:02,065][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:42:02,391][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:42:02,718][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:42:03,043][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:42:03,369][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:42:03,695][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:42:04,020][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:42:04,346][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:42:04,671][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:42:04,996][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:42:05,738][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:42:06,496][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:06,498][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:06,500][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:07,484][__main__][INFO] - Iteration 109 took 20s (31.65% Gen, 63.63% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 45m 45s. Estimated total time: 17h 24m 4s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 48s, 500 more iterations: 2h 54m 0s.
[2025-11-13 08:42:07,486][__main__][INFO] - Starting iteration 109.
[2025-11-13 08:42:07,489][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:42:07,490][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:14,161][__main__][INFO] - Number of regex retries in iteration 109: 0
[2025-11-13 08:42:14,161][__main__][INFO] - agents played in iteration 109 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:42:14,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:14,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:14,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:14,761][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:14,762][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:14,762][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:15,555][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:42:15,853][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:42:16,179][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:42:16,506][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:42:16,833][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:42:17,160][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:42:17,490][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:42:17,816][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:42:18,143][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:42:18,468][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:42:18,796][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:42:19,125][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:42:19,452][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:42:19,778][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:42:20,110][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:42:20,436][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:42:20,764][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:42:21,090][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:42:21,419][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:42:21,745][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:42:22,072][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:42:22,398][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:42:22,728][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:42:23,051][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:42:23,378][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:42:23,704][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:42:24,032][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:42:24,357][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:42:24,683][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:42:25,010][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:42:25,336][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:42:25,662][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:42:25,989][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:42:26,755][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:42:27,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:27,529][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:27,531][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:28,582][__main__][INFO] - Iteration 110 took 21s (31.63% Gen, 63.39% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 56m 0s. Estimated total time: 17h 34m 40s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 9s, 500 more iterations: 2h 55m 46s.
[2025-11-13 08:42:28,584][__main__][INFO] - Starting iteration 110.
[2025-11-13 08:42:28,586][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 10 and human policies 1.
[2025-11-13 08:42:28,587][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:35,271][__main__][INFO] - Number of regex retries in iteration 110: 0
[2025-11-13 08:42:35,272][__main__][INFO] - agents played in iteration 110 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:42:35,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:35,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:35,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:35,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:35,874][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:35,875][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:36,652][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:42:36,951][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:42:37,276][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:42:37,602][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:42:37,929][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:42:38,254][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:42:38,583][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:42:38,910][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:42:39,236][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:42:39,563][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:42:39,889][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:42:40,218][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:42:40,546][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:42:40,873][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:42:41,199][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:42:41,524][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:42:41,850][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:42:42,176][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:42:42,502][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:42:42,828][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:42:43,154][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:42:43,480][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:42:43,809][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:42:44,138][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:42:44,467][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:42:44,794][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:42:45,127][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:42:45,452][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:42:45,779][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:42:46,107][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:42:46,436][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:42:46,764][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:42:47,093][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:42:47,839][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:42:48,592][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:42:48,594][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:42:48,595][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:42:50,553][__main__][INFO] - Iteration 111 took 21s (30.43% Gen, 60.65% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 39m 21s. Estimated total time: 18h 18m 23s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 36s, 500 more iterations: 3h 3m 3s.
[2025-11-13 08:42:50,555][__main__][INFO] - Starting iteration 111.
[2025-11-13 08:42:50,558][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:42:50,559][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:42:57,587][__main__][INFO] - Number of regex retries in iteration 111: 0
[2025-11-13 08:42:57,588][__main__][INFO] - agents played in iteration 111 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:42:58,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,153][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:42:58,194][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:42:58,195][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:42:58,980][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:42:59,279][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:42:59,605][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:42:59,930][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:43:00,256][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:43:00,583][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:43:00,910][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:43:01,237][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:43:01,564][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:43:01,890][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:43:02,222][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:43:02,548][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:43:02,875][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:43:03,203][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:43:03,528][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:43:03,854][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:43:04,180][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:43:04,506][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:43:04,833][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:43:05,160][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:43:05,486][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:43:05,813][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:43:06,139][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:43:06,465][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:43:06,791][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:43:07,118][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:43:07,450][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:43:07,770][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:43:08,097][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:43:08,424][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:43:08,749][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:43:09,077][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:43:09,407][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:43:10,173][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:43:10,948][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:10,950][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:10,952][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:11,976][__main__][INFO] - Iteration 112 took 21s (32.82% Gen, 62.40% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 11m 32s. Estimated total time: 17h 50m 55s. Time estimates for 10 more iterations: 3m 34s, 100 more iterations: 35m 41s, 500 more iterations: 2h 58m 29s.
[2025-11-13 08:43:11,978][__main__][INFO] - Starting iteration 112.
[2025-11-13 08:43:11,982][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:43:11,982][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:43:18,847][__main__][INFO] - Number of regex retries in iteration 112: 0
[2025-11-13 08:43:18,848][__main__][INFO] - agents played in iteration 112 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:43:19,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,357][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:19,439][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:43:19,439][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:43:20,219][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:43:20,518][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:43:20,844][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:43:21,171][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:43:21,499][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:43:21,825][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:43:22,154][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:43:22,483][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:43:22,810][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:43:23,136][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:43:23,463][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:43:23,789][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:43:24,115][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:43:24,446][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:43:24,768][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:43:25,094][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:43:25,421][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:43:25,751][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:43:26,076][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:43:26,401][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:43:26,727][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:43:27,058][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:43:27,380][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:43:27,706][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:43:28,033][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:43:28,367][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:43:28,693][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:43:29,019][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:43:29,345][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:43:29,671][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:43:29,998][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:43:30,323][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:43:30,651][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:43:31,436][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:43:32,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:32,204][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:32,206][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:33,235][__main__][INFO] - Iteration 113 took 21s (32.30% Gen, 62.85% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 2m 59s. Estimated total time: 17h 42m 43s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 25s, 500 more iterations: 2h 57m 7s.
[2025-11-13 08:43:33,238][__main__][INFO] - Starting iteration 113.
[2025-11-13 08:43:33,241][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:43:33,242][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:43:40,054][__main__][INFO] - Number of regex retries in iteration 113: 0
[2025-11-13 08:43:40,055][__main__][INFO] - agents played in iteration 113 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:43:40,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,594][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:43:40,635][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:43:40,636][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:43:41,388][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:43:41,686][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:43:42,013][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:43:42,340][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:43:42,665][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:43:42,990][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:43:43,317][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:43:43,642][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:43:43,967][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:43:44,293][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:43:44,618][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:43:44,946][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:43:45,274][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:43:45,603][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:43:45,931][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:43:46,258][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:43:46,588][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:43:46,915][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:43:47,241][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:43:47,568][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:43:47,896][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:43:48,226][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:43:48,554][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:43:48,881][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:43:49,209][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:43:49,540][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:43:49,867][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:43:50,196][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:43:50,521][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:43:50,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:43:51,174][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:43:51,502][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:43:51,828][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:43:52,582][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:43:53,344][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:43:53,345][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:43:53,347][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:43:54,353][__main__][INFO] - Iteration 114 took 21s (32.27% Gen, 62.96% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 55m 32s. Estimated total time: 17h 35m 38s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 11s, 500 more iterations: 2h 55m 56s.
[2025-11-13 08:43:54,355][__main__][INFO] - Starting iteration 114.
[2025-11-13 08:43:54,358][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:43:54,359][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:01,349][__main__][INFO] - Number of regex retries in iteration 114: 0
[2025-11-13 08:44:01,350][__main__][INFO] - agents played in iteration 114 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:44:01,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:01,870][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:01,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:01,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:01,952][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:01,952][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:02,730][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:44:03,028][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:44:03,361][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:44:03,694][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:44:04,029][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:44:04,358][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:44:04,691][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:44:05,019][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:44:05,357][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:44:05,683][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:44:06,009][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:44:06,339][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:44:06,673][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:44:07,003][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:44:07,332][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:44:07,659][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:44:07,988][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:44:08,317][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:44:08,644][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:44:08,976][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:44:09,314][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:44:09,644][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:44:09,974][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:44:10,308][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:44:10,649][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:44:10,978][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:44:11,307][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:44:11,637][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:44:11,965][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:44:12,290][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:44:12,620][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:44:12,946][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:44:13,273][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:44:14,036][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:44:14,792][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:14,794][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:14,795][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:15,990][__main__][INFO] - Iteration 115 took 21s (32.32% Gen, 62.15% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 21m 11s. Estimated total time: 18h 1m 38s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 3s, 500 more iterations: 3h 0m 16s.
[2025-11-13 08:44:15,992][__main__][INFO] - Starting iteration 115.
[2025-11-13 08:44:15,996][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:44:15,997][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:22,847][__main__][INFO] - Number of regex retries in iteration 115: 0
[2025-11-13 08:44:22,848][__main__][INFO] - agents played in iteration 115 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:44:23,313][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,353][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:23,434][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:23,434][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:24,250][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:44:24,547][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:44:24,881][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:44:25,208][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:44:25,536][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:44:25,863][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:44:26,189][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:44:26,515][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:44:26,841][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:44:27,166][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:44:27,497][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:44:27,826][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:44:28,158][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:44:28,487][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:44:28,815][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:44:29,142][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:44:29,471][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:44:29,800][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:44:30,126][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:44:30,454][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:44:30,783][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:44:31,109][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:44:31,438][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:44:31,770][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:44:32,104][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:44:32,437][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:44:32,769][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:44:33,098][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:44:33,432][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:44:33,761][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:44:34,089][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:44:34,420][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:44:34,754][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:44:35,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:44:36,293][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:36,295][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:36,296][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:37,362][__main__][INFO] - Iteration 116 took 21s (32.06% Gen, 62.94% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 7m 30s. Estimated total time: 17h 48m 19s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 36s, 500 more iterations: 2h 58m 3s.
[2025-11-13 08:44:37,364][__main__][INFO] - Starting iteration 116.
[2025-11-13 08:44:37,367][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:44:37,367][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:44:44,247][__main__][INFO] - Number of regex retries in iteration 116: 0
[2025-11-13 08:44:44,248][__main__][INFO] - agents played in iteration 116 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:44:44,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:44:44,858][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:44:44,859][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:44:45,629][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:44:45,927][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:44:46,253][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:44:46,581][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:44:46,908][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:44:47,234][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:44:47,561][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:44:47,887][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:44:48,213][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:44:48,548][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:44:48,875][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:44:49,204][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:44:49,531][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:44:49,859][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:44:50,186][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:44:50,518][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:44:50,844][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:44:51,171][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:44:51,498][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:44:51,825][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:44:52,151][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:44:52,479][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:44:52,807][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:44:53,133][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:44:53,459][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:44:53,786][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:44:54,113][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:44:54,440][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:44:54,765][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:44:55,093][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:44:55,424][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:44:55,753][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:44:56,080][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:44:56,831][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:44:57,587][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:44:57,588][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:44:57,590][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:44:58,570][__main__][INFO] - Iteration 117 took 21s (32.45% Gen, 62.92% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 59m 4s. Estimated total time: 17h 40m 13s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 20s, 500 more iterations: 2h 56m 42s.
[2025-11-13 08:44:58,573][__main__][INFO] - Starting iteration 117.
[2025-11-13 08:44:58,576][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:44:58,576][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:05,459][__main__][INFO] - Number of regex retries in iteration 117: 0
[2025-11-13 08:45:05,459][__main__][INFO] - agents played in iteration 117 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:45:05,931][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:05,972][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,012][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:06,052][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:06,053][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:06,842][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:45:07,150][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:45:07,476][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:45:07,803][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:45:08,129][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:45:08,454][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:45:08,781][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:45:09,108][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:45:09,438][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:45:09,771][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:45:10,101][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:45:10,428][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:45:10,753][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:45:11,080][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:45:11,406][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:45:11,733][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:45:12,059][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:45:12,385][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:45:12,711][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:45:13,039][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:45:13,365][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:45:13,692][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:45:14,018][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:45:14,346][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:45:14,678][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:45:15,004][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:45:15,332][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:45:15,657][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:45:15,986][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:45:16,312][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:45:16,638][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:45:16,964][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:45:17,291][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:45:18,063][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:45:18,834][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:45:18,836][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:45:18,837][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:45:19,893][__main__][INFO] - Iteration 118 took 21s (32.29% Gen, 62.75% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 4m 24s. Estimated total time: 17h 45m 55s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 31s, 500 more iterations: 2h 57m 39s.
[2025-11-13 08:45:19,896][__main__][INFO] - Starting iteration 118.
[2025-11-13 08:45:19,899][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:45:19,900][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:26,783][__main__][INFO] - Number of regex retries in iteration 118: 0
[2025-11-13 08:45:26,784][__main__][INFO] - agents played in iteration 118 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:45:27,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,293][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,333][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:27,373][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:27,374][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:28,153][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:45:28,451][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:45:28,778][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:45:29,109][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:45:29,446][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:45:29,778][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:45:30,111][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:45:30,440][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:45:30,766][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:45:31,095][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:45:31,421][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:45:31,746][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:45:32,075][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:45:32,401][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:45:32,729][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:45:33,058][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:45:33,381][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:45:33,713][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:45:34,039][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:45:34,365][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:45:34,692][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:45:35,019][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:45:35,348][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:45:35,678][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:45:36,003][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:45:36,330][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:45:36,657][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:45:36,983][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:45:37,308][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:45:37,633][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:45:37,962][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:45:38,290][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:45:38,618][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:45:39,364][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:45:40,118][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:45:40,119][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:45:40,121][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:45:41,107][__main__][INFO] - Iteration 119 took 21s (32.46% Gen, 62.88% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 58m 33s. Estimated total time: 17h 40m 26s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 20s, 500 more iterations: 2h 56m 44s.
[2025-11-13 08:45:41,108][__main__][INFO] - Starting iteration 119.
[2025-11-13 08:45:41,111][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:45:41,111][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:45:47,698][__main__][INFO] - Number of regex retries in iteration 119: 0
[2025-11-13 08:45:47,699][__main__][INFO] - agents played in iteration 119 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:45:48,162][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,202][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:45:48,284][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:45:48,284][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:45:49,067][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:45:49,365][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:45:49,691][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:45:50,017][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:45:50,343][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:45:50,671][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:45:50,997][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:45:51,324][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:45:51,650][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:45:51,977][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:45:52,303][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:45:52,632][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:45:52,972][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:45:53,298][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:45:53,625][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:45:53,952][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:45:54,278][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:45:54,604][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:45:54,937][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:45:55,270][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:45:55,600][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:45:55,926][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:45:56,256][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:45:56,588][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:45:56,917][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:45:57,243][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:45:57,573][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:45:57,907][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:45:58,235][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:45:58,564][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:45:58,894][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:45:59,224][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:45:59,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:46:00,333][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:46:01,091][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:46:01,093][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:46:01,095][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:46:02,095][__main__][INFO] - Iteration 120 took 20s (31.39% Gen, 63.84% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 47m 2s. Estimated total time: 17h 29m 15s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 58s, 500 more iterations: 2h 54m 52s.
[2025-11-13 08:46:02,098][__main__][INFO] - Starting iteration 120.
[2025-11-13 08:46:02,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 11 and human policies 1.
[2025-11-13 08:46:02,101][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:09,143][__main__][INFO] - Number of regex retries in iteration 120: 0
[2025-11-13 08:46:09,143][__main__][INFO] - agents played in iteration 120 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:46:09,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,652][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,733][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:09,734][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:09,734][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:10,523][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:46:10,819][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:46:11,146][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:46:11,474][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:46:11,801][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:46:12,126][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:46:12,451][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:46:12,778][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:46:13,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:46:13,433][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:46:13,760][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:46:14,088][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:46:14,415][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:46:14,741][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:46:15,068][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:46:15,394][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:46:15,720][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:46:16,046][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:46:16,374][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:46:16,700][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:46:17,026][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:46:17,353][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:46:17,680][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:46:18,006][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:46:18,333][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:46:18,660][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:46:18,987][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:46:19,313][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:46:19,640][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:46:19,967][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:46:20,296][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:46:20,620][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:46:20,947][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:46:21,712][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:46:22,462][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:46:22,464][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:46:22,466][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:46:24,434][__main__][INFO] - Iteration 121 took 22s (31.53% Gen, 59.65% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 54m 7s. Estimated total time: 18h 36m 43s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 13s, 500 more iterations: 3h 6m 7s.
[2025-11-13 08:46:24,436][__main__][INFO] - Starting iteration 121.
[2025-11-13 08:46:24,440][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:46:24,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:31,706][__main__][INFO] - Number of regex retries in iteration 121: 0
[2025-11-13 08:46:31,707][__main__][INFO] - agents played in iteration 121 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:46:32,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,220][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,300][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:32,301][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:32,301][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:33,102][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:46:33,401][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:46:33,731][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:46:34,058][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:46:34,387][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:46:34,720][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:46:35,049][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:46:35,378][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:46:35,713][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:46:36,039][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:46:36,380][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:46:36,707][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:46:37,037][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:46:37,364][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:46:37,693][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:46:38,020][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:46:38,347][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:46:38,675][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:46:39,003][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:46:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:46:39,657][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:46:39,989][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:46:40,311][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:46:40,640][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:46:40,968][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:46:41,298][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:46:41,626][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:46:41,953][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:46:42,280][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:46:42,606][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:46:42,932][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:46:43,259][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:46:43,586][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:46:44,372][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:46:45,148][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:46:45,150][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:46:45,151][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:46:46,172][__main__][INFO] - Iteration 122 took 21s (33.43% Gen, 61.86% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 23m 41s. Estimated total time: 18h 6m 38s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 13s, 500 more iterations: 3h 1m 6s.
[2025-11-13 08:46:46,174][__main__][INFO] - Starting iteration 122.
[2025-11-13 08:46:46,178][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:46:46,178][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:46:53,463][__main__][INFO] - Number of regex retries in iteration 122: 0
[2025-11-13 08:46:53,464][__main__][INFO] - agents played in iteration 122 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:46:53,931][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:53,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:46:54,051][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:46:54,052][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:46:54,854][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:46:55,151][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:46:55,478][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:46:55,803][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:46:56,133][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:46:56,459][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:46:56,786][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:46:57,112][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:46:57,439][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:46:57,765][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:46:58,092][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:46:58,418][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:46:58,744][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:46:59,071][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:46:59,400][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:46:59,727][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:47:00,054][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:47:00,382][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:47:00,708][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:47:01,034][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:47:01,360][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:47:01,686][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:47:02,012][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:47:02,338][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:47:02,669][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:47:02,995][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:47:03,323][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:47:03,649][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:47:03,976][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:47:04,301][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:47:04,627][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:47:04,954][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:47:05,280][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:47:06,051][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:47:06,845][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:06,846][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:06,848][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:07,864][__main__][INFO] - Iteration 123 took 21s (33.59% Gen, 61.72% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 21m 2s. Estimated total time: 18h 4m 21s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 8s, 500 more iterations: 3h 0m 43s.
[2025-11-13 08:47:07,866][__main__][INFO] - Starting iteration 123.
[2025-11-13 08:47:07,869][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:47:07,870][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:14,874][__main__][INFO] - Number of regex retries in iteration 123: 0
[2025-11-13 08:47:14,875][__main__][INFO] - agents played in iteration 123 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:47:15,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,377][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,418][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,458][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:15,459][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:15,459][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:16,235][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:47:16,534][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:47:16,862][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:47:17,189][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:47:17,518][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:47:17,845][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:47:18,177][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:47:18,503][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:47:18,829][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:47:19,155][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:47:19,483][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:47:19,809][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:47:20,134][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:47:20,461][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:47:20,787][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:47:21,116][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:47:21,443][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:47:21,768][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:47:22,097][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:47:22,424][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:47:22,750][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:47:23,076][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:47:23,406][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:47:23,734][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:47:24,060][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:47:24,389][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:47:24,715][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:47:25,042][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:47:25,368][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:47:25,698][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:47:26,023][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:47:26,350][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:47:26,676][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:47:27,413][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:47:28,171][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:28,173][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:28,174][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:29,183][__main__][INFO] - Iteration 124 took 21s (32.87% Gen, 62.40% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 2m 3s. Estimated total time: 17h 45m 43s. Time estimates for 10 more iterations: 3m 33s, 100 more iterations: 35m 31s, 500 more iterations: 2h 57m 37s.
[2025-11-13 08:47:29,185][__main__][INFO] - Starting iteration 124.
[2025-11-13 08:47:29,188][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:47:29,189][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:36,486][__main__][INFO] - Number of regex retries in iteration 124: 0
[2025-11-13 08:47:36,488][__main__][INFO] - agents played in iteration 124 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:47:36,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:36,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,070][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:37,071][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:37,071][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:37,833][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:47:38,132][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:47:38,459][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:47:38,786][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:47:39,114][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:47:39,439][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:47:39,765][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:47:40,094][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:47:40,420][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:47:40,746][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:47:41,073][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:47:41,400][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:47:41,726][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:47:42,052][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:47:42,379][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:47:42,705][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:47:43,031][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:47:43,358][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:47:43,685][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:47:44,011][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:47:44,337][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:47:44,664][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:47:44,991][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:47:45,318][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:47:45,644][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:47:45,971][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:47:46,299][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:47:46,625][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:47:46,951][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:47:47,278][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:47:47,605][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:47:47,932][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:47:48,258][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:47:48,984][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:47:49,745][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:47:49,747][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:47:49,748][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:47:50,913][__main__][INFO] - Iteration 125 took 21s (33.60% Gen, 61.03% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 22m 16s. Estimated total time: 18h 6m 18s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 12s, 500 more iterations: 3h 1m 3s.
[2025-11-13 08:47:50,915][__main__][INFO] - Starting iteration 125.
[2025-11-13 08:47:50,920][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:47:50,920][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:47:58,207][__main__][INFO] - Number of regex retries in iteration 125: 0
[2025-11-13 08:47:58,208][__main__][INFO] - agents played in iteration 125 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:47:58,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:47:58,817][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:47:58,817][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:47:59,598][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:47:59,895][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:48:00,222][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:48:00,548][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:48:00,874][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:48:01,201][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:48:01,527][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:48:01,854][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:48:02,180][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:48:02,506][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:48:02,833][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:48:03,159][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:48:03,486][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:48:03,812][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:48:04,138][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:48:04,468][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:48:04,794][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:48:05,121][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:48:05,447][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:48:05,774][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:48:06,101][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:48:06,433][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:48:06,763][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:48:07,089][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:48:07,418][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:48:07,744][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:48:08,071][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:48:08,399][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:48:08,727][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:48:09,053][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:48:09,384][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:48:09,709][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:48:10,036][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:48:10,803][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:48:11,569][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:11,571][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:11,573][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:12,576][__main__][INFO] - Iteration 126 took 21s (33.65% Gen, 61.71% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 18m 27s. Estimated total time: 18h 2m 51s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 5s, 500 more iterations: 3h 0m 28s.
[2025-11-13 08:48:12,578][__main__][INFO] - Starting iteration 126.
[2025-11-13 08:48:12,582][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:48:12,582][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:48:19,733][__main__][INFO] - Number of regex retries in iteration 126: 0
[2025-11-13 08:48:19,733][__main__][INFO] - agents played in iteration 126 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:48:20,198][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:20,324][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:48:20,324][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:48:21,080][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:48:21,378][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:48:21,705][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:48:22,031][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:48:22,357][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:48:22,683][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:48:23,009][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:48:23,336][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:48:23,663][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:48:23,990][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:48:24,316][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:48:24,647][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:48:24,973][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:48:25,300][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:48:25,628][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:48:25,956][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:48:26,283][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:48:26,613][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:48:26,942][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:48:27,269][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:48:27,599][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:48:27,928][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:48:28,254][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:48:28,580][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:48:28,906][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:48:29,233][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:48:29,559][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:48:29,887][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:48:30,214][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:48:30,540][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:48:30,866][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:48:31,194][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:48:31,522][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:48:32,242][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:48:33,001][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:33,002][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:33,004][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:33,995][__main__][INFO] - Iteration 127 took 21s (33.39% Gen, 61.97% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 5m 56s. Estimated total time: 17h 50m 41s. Time estimates for 10 more iterations: 3m 34s, 100 more iterations: 35m 41s, 500 more iterations: 2h 58m 26s.
[2025-11-13 08:48:33,998][__main__][INFO] - Starting iteration 127.
[2025-11-13 08:48:34,006][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:48:34,007][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:48:41,120][__main__][INFO] - Number of regex retries in iteration 127: 0
[2025-11-13 08:48:41,120][__main__][INFO] - agents played in iteration 127 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:48:41,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:41,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:41,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:41,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:48:41,701][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:48:41,701][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:48:42,501][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:48:42,799][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:48:43,127][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:48:43,455][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:48:43,781][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:48:44,115][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:48:44,442][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:48:44,775][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:48:45,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:48:45,437][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:48:45,771][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:48:46,106][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:48:46,439][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:48:46,768][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:48:47,095][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:48:47,426][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:48:47,754][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:48:48,083][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:48:48,410][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:48:48,735][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:48:49,062][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:48:49,390][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:48:49,722][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:48:50,044][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:48:50,372][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:48:50,698][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:48:51,030][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:48:51,353][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:48:51,680][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:48:52,007][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:48:52,337][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:48:52,662][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:48:52,989][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:48:53,722][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:48:54,477][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:48:54,479][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:48:54,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:48:55,756][__main__][INFO] - Iteration 128 took 21s (32.69% Gen, 61.42% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 22m 32s. Estimated total time: 18h 7m 38s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 15s, 500 more iterations: 3h 1m 16s.
[2025-11-13 08:48:55,758][__main__][INFO] - Starting iteration 128.
[2025-11-13 08:48:55,762][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:48:55,763][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:02,983][__main__][INFO] - Number of regex retries in iteration 128: 0
[2025-11-13 08:49:02,983][__main__][INFO] - agents played in iteration 128 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:49:03,451][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,492][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:03,574][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:03,575][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:04,370][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:49:04,669][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:49:04,995][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:49:05,321][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:49:05,647][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:49:05,974][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:49:06,301][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:49:06,627][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:49:06,954][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:49:07,280][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:49:07,605][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:49:07,933][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:49:08,260][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:49:08,587][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:49:08,913][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:49:09,241][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:49:09,566][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:49:09,895][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:49:10,220][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:49:10,547][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:49:10,873][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:49:11,200][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:49:11,526][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:49:11,853][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:49:12,180][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:49:12,507][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:49:12,833][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:49:13,160][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:49:13,487][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:49:13,817][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:49:14,143][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:49:14,471][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:49:14,801][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:49:15,572][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:49:16,369][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:16,371][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:16,372][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:49:17,709][__main__][INFO] - Iteration 129 took 21s (32.90% Gen, 61.00% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 31m 54s. Estimated total time: 18h 17m 23s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 34s, 500 more iterations: 3h 2m 53s.
[2025-11-13 08:49:17,711][__main__][INFO] - Starting iteration 129.
[2025-11-13 08:49:17,714][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:49:17,714][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:24,807][__main__][INFO] - Number of regex retries in iteration 129: 0
[2025-11-13 08:49:24,807][__main__][INFO] - agents played in iteration 129 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:49:25,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:25,410][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:25,410][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:26,184][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:49:26,482][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:49:26,809][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:49:27,137][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:49:27,463][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:49:27,789][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:49:28,121][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:49:28,447][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:49:28,774][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:49:29,102][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:49:29,429][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:49:29,754][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:49:30,080][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:49:30,407][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:49:30,734][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:49:31,060][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:49:31,386][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:49:31,713][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:49:32,039][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:49:32,366][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:49:32,692][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:49:33,018][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:49:33,344][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:49:33,671][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:49:33,997][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:49:34,324][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:49:34,650][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:49:34,976][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:49:35,303][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:49:35,629][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:49:35,956][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:49:36,283][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:49:36,613][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:49:37,367][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:49:38,119][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:38,121][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:38,122][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:49:39,283][__main__][INFO] - Iteration 130 took 21s (32.88% Gen, 61.73% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 12m 38s. Estimated total time: 17h 58m 29s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 56s, 500 more iterations: 2h 59m 44s.
[2025-11-13 08:49:39,285][__main__][INFO] - Starting iteration 130.
[2025-11-13 08:49:39,287][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 12 and human policies 1.
[2025-11-13 08:49:39,288][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:49:46,071][__main__][INFO] - Number of regex retries in iteration 130: 0
[2025-11-13 08:49:46,071][__main__][INFO] - agents played in iteration 130 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:49:46,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:49:46,660][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:49:46,661][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:49:47,439][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:49:47,737][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:49:48,069][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:49:48,396][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:49:48,725][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:49:49,051][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:49:49,377][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:49:49,702][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:49:50,027][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:49:50,353][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:49:50,680][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:49:51,006][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:49:51,332][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:49:51,661][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:49:51,987][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:49:52,314][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:49:52,640][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:49:52,967][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:49:53,293][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:49:53,622][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:49:53,950][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:49:54,275][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:49:54,603][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:49:54,932][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:49:55,259][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:49:55,585][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:49:55,915][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:49:56,243][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:49:56,568][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:49:56,895][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:49:57,220][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:49:57,546][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:49:57,873][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:49:58,623][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:49:59,358][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:49:59,360][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:49:59,362][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:01,386][__main__][INFO] - Iteration 131 took 22s (30.69% Gen, 60.14% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 38m 47s. Estimated total time: 18h 24m 59s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 49s, 500 more iterations: 3h 4m 9s.
[2025-11-13 08:50:01,388][__main__][INFO] - Starting iteration 131.
[2025-11-13 08:50:01,392][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:50:01,392][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:08,996][__main__][INFO] - Number of regex retries in iteration 131: 0
[2025-11-13 08:50:08,996][__main__][INFO] - agents played in iteration 131 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:50:09,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,492][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:09,573][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:09,573][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:10,328][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:50:10,627][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:50:10,956][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:50:11,284][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:50:11,619][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:50:11,945][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:50:12,274][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:50:12,602][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:50:12,932][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:50:13,258][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:50:13,586][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:50:13,914][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:50:14,236][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:50:14,565][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:50:14,894][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:50:15,226][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:50:15,545][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:50:15,874][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:50:16,207][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:50:16,542][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:50:16,868][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:50:17,193][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:50:17,519][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:50:17,845][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:50:18,173][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:50:18,500][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:50:18,825][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:50:19,151][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:50:19,478][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:50:19,812][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:50:20,141][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:50:20,466][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:50:20,800][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:50:21,543][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:50:22,302][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:50:22,304][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:50:22,306][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:23,323][__main__][INFO] - Iteration 132 took 21s (34.67% Gen, 60.69% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 30m 1s. Estimated total time: 18h 16m 36s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 33s, 500 more iterations: 3h 2m 46s.
[2025-11-13 08:50:23,325][__main__][INFO] - Starting iteration 132.
[2025-11-13 08:50:23,329][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:50:23,329][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:30,666][__main__][INFO] - Number of regex retries in iteration 132: 0
[2025-11-13 08:50:30,667][__main__][INFO] - agents played in iteration 132 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:50:31,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:31,252][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:31,253][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:32,018][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:50:32,321][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:50:32,643][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:50:32,970][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:50:33,298][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:50:33,631][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:50:33,953][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:50:34,280][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:50:34,607][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:50:34,937][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:50:35,259][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:50:35,585][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:50:35,911][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:50:36,240][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:50:36,570][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:50:36,896][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:50:37,223][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:50:37,549][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:50:37,880][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:50:38,206][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:50:38,531][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:50:38,858][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:50:39,186][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:50:39,513][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:50:39,839][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:50:40,165][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:50:40,491][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:50:40,819][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:50:41,145][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:50:41,471][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:50:41,798][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:50:42,129][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:50:42,455][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:50:43,190][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:50:43,944][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:50:43,946][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:50:43,948][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:50:45,240][__main__][INFO] - Iteration 133 took 21s (33.49% Gen, 60.61% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 28m 42s. Estimated total time: 18h 15m 38s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 31s, 500 more iterations: 3h 2m 36s.
[2025-11-13 08:50:45,243][__main__][INFO] - Starting iteration 133.
[2025-11-13 08:50:45,246][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:50:45,247][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:50:52,680][__main__][INFO] - Number of regex retries in iteration 133: 0
[2025-11-13 08:50:52,681][__main__][INFO] - agents played in iteration 133 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:50:53,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,197][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,239][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:50:53,281][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:50:53,281][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:50:54,076][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:50:54,374][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:50:54,702][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:50:55,028][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:50:55,354][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:50:55,680][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:50:56,006][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:50:56,333][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:50:56,660][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:50:56,986][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:50:57,312][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:50:57,638][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:50:57,967][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:50:58,292][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:50:58,618][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:50:58,944][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:50:59,270][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:50:59,595][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:50:59,921][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:51:00,247][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:51:00,573][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:51:00,898][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:51:01,224][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:51:01,550][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:51:01,876][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:51:02,206][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:51:02,538][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:51:02,864][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:51:03,191][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:51:03,516][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:51:03,842][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:51:04,168][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:51:04,494][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:51:05,256][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:51:06,011][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:06,013][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:06,015][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:07,022][__main__][INFO] - Iteration 134 took 21s (34.14% Gen, 61.23% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 21m 33s. Estimated total time: 18h 8m 51s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 17s, 500 more iterations: 3h 1m 28s.
[2025-11-13 08:51:07,024][__main__][INFO] - Starting iteration 134.
[2025-11-13 08:51:07,028][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:51:07,028][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:14,612][__main__][INFO] - Number of regex retries in iteration 134: 0
[2025-11-13 08:51:14,613][__main__][INFO] - agents played in iteration 134 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:51:15,091][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:15,211][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:51:15,212][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:51:15,996][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:51:16,293][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:51:16,620][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:51:16,947][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:51:17,273][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:51:17,603][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:51:17,930][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:51:18,255][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:51:18,581][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:51:18,908][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:51:19,234][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:51:19,559][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:51:19,885][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:51:20,212][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:51:20,538][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:51:20,864][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:51:21,191][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:51:21,519][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:51:21,845][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:51:22,171][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:51:22,497][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:51:22,825][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:51:23,152][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:51:23,479][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:51:23,806][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:51:24,132][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:51:24,457][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:51:24,789][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:51:25,118][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:51:25,443][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:51:25,769][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:51:26,095][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:51:26,434][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:51:27,174][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:51:27,945][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:27,947][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:27,948][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:29,085][__main__][INFO] - Iteration 135 took 22s (34.38% Gen, 60.46% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 35m 15s. Estimated total time: 18h 22m 55s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 45s, 500 more iterations: 3h 3m 49s.
[2025-11-13 08:51:29,088][__main__][INFO] - Starting iteration 135.
[2025-11-13 08:51:29,090][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:51:29,091][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:36,583][__main__][INFO] - Number of regex retries in iteration 135: 0
[2025-11-13 08:51:36,584][__main__][INFO] - agents played in iteration 135 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:51:37,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,162][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:37,203][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:51:37,204][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:51:37,977][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:51:38,275][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:51:38,603][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:51:38,929][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:51:39,256][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:51:39,594][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:51:39,928][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:51:40,258][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:51:40,589][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:51:40,920][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:51:41,252][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:51:41,582][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:51:41,908][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:51:42,233][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:51:42,560][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:51:42,886][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:51:43,213][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:51:43,544][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:51:43,870][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:51:44,197][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:51:44,523][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:51:44,849][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:51:45,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:51:45,502][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:51:45,829][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:51:46,161][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:51:46,488][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:51:46,815][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:51:47,141][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:51:47,468][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:51:47,794][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:51:48,120][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:51:48,451][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:51:49,202][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:51:49,945][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:51:49,947][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:51:49,949][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:51:50,930][__main__][INFO] - Iteration 136 took 21s (34.30% Gen, 61.20% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 23m 59s. Estimated total time: 18h 12m 1s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 24s, 500 more iterations: 3h 2m 0s.
[2025-11-13 08:51:50,932][__main__][INFO] - Starting iteration 136.
[2025-11-13 08:51:50,934][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:51:50,935][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:51:58,417][__main__][INFO] - Number of regex retries in iteration 136: 0
[2025-11-13 08:51:58,418][__main__][INFO] - agents played in iteration 136 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:51:58,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,914][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:51:58,995][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:51:58,995][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:51:59,748][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:52:00,044][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:52:00,370][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:52:00,700][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:52:01,030][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:52:01,357][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:52:01,684][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:52:02,023][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:52:02,352][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:52:02,679][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:52:03,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:52:03,339][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:52:03,668][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:52:03,996][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:52:04,326][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:52:04,658][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:52:04,988][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:52:05,313][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:52:05,644][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:52:05,966][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:52:06,291][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:52:06,617][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:52:06,949][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:52:07,278][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:52:07,604][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:52:07,930][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:52:08,263][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:52:08,589][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:52:08,914][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:52:09,240][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:52:09,568][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:52:09,894][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:52:10,222][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:52:10,961][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:52:11,711][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:11,713][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:11,715][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:12,711][__main__][INFO] - Iteration 137 took 21s (34.36% Gen, 61.06% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 20m 28s. Estimated total time: 18h 8m 52s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 17s, 500 more iterations: 3h 1m 28s.
[2025-11-13 08:52:12,713][__main__][INFO] - Starting iteration 137.
[2025-11-13 08:52:12,717][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:52:12,717][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:52:20,190][__main__][INFO] - Number of regex retries in iteration 137: 0
[2025-11-13 08:52:20,190][__main__][INFO] - agents played in iteration 137 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:52:20,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:20,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:20,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:20,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:20,779][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:52:20,780][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:52:21,528][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:52:21,840][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:52:22,167][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:52:22,492][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:52:22,817][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:52:23,143][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:52:23,471][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:52:23,798][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:52:24,130][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:52:24,459][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:52:24,788][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:52:25,121][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:52:25,449][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:52:25,777][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:52:26,102][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:52:26,428][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:52:26,755][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:52:27,081][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:52:27,408][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:52:27,737][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:52:28,063][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:52:28,389][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:52:28,718][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:52:29,046][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:52:29,374][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:52:29,709][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:52:30,039][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:52:30,364][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:52:30,690][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:52:31,027][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:52:31,354][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:52:31,681][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:52:32,006][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:52:32,767][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:52:33,529][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:33,531][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:33,533][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:34,530][__main__][INFO] - Iteration 138 took 21s (34.26% Gen, 61.16% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 21m 57s. Estimated total time: 18h 10m 43s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 21s, 500 more iterations: 3h 1m 47s.
[2025-11-13 08:52:34,532][__main__][INFO] - Starting iteration 138.
[2025-11-13 08:52:34,536][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:52:34,536][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:52:42,061][__main__][INFO] - Number of regex retries in iteration 138: 0
[2025-11-13 08:52:42,062][__main__][INFO] - agents played in iteration 138 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:52:42,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:42,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:42,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:42,660][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:52:42,661][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:52:42,661][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:52:43,449][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:52:43,747][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:52:44,075][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:52:44,402][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:52:44,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:52:45,055][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:52:45,382][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:52:45,709][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:52:46,035][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:52:46,362][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:52:46,688][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:52:47,014][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:52:47,341][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:52:47,667][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:52:47,993][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:52:48,320][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:52:48,646][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:52:48,974][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:52:49,299][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:52:49,626][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:52:49,953][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:52:50,278][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:52:50,605][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:52:50,945][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:52:51,273][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:52:51,600][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:52:51,927][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:52:52,254][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:52:52,580][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:52:52,906][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:52:53,233][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:52:53,558][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:52:53,887][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:52:54,636][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:52:55,405][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:52:55,407][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:52:55,409][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:52:56,410][__main__][INFO] - Iteration 139 took 21s (34.40% Gen, 61.01% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 24m 37s. Estimated total time: 18h 13m 45s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 27s, 500 more iterations: 3h 2m 17s.
[2025-11-13 08:52:56,412][__main__][INFO] - Starting iteration 139.
[2025-11-13 08:52:56,416][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:52:56,417][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:53:03,959][__main__][INFO] - Number of regex retries in iteration 139: 0
[2025-11-13 08:53:03,960][__main__][INFO] - agents played in iteration 139 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:53:04,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,510][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:04,552][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:04,552][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:05,350][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:53:05,648][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:53:05,977][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:53:06,309][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:53:06,649][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:53:06,976][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:53:07,309][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:53:07,638][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:53:07,968][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:53:08,298][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:53:08,627][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:53:08,957][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:53:09,287][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:53:09,618][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:53:09,945][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:53:10,277][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:53:10,613][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:53:10,948][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:53:11,281][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:53:11,611][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:53:11,944][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:53:12,278][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:53:12,608][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:53:12,934][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:53:13,266][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:53:13,599][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:53:13,929][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:53:14,262][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:53:14,591][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:53:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:53:15,249][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:53:15,582][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:53:15,919][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:53:16,679][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:53:17,466][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:53:17,468][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:53:17,470][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:53:18,529][__main__][INFO] - Iteration 140 took 22s (34.11% Gen, 61.09% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 36m 12s. Estimated total time: 18h 25m 41s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 51s, 500 more iterations: 3h 4m 16s.
[2025-11-13 08:53:18,531][__main__][INFO] - Starting iteration 140.
[2025-11-13 08:53:18,537][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 13 and human policies 1.
[2025-11-13 08:53:18,538][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:53:26,032][__main__][INFO] - Number of regex retries in iteration 140: 0
[2025-11-13 08:53:26,033][__main__][INFO] - agents played in iteration 140 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:53:26,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:26,615][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:26,615][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:27,383][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:53:27,680][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:53:28,006][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:53:28,330][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:53:28,660][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:53:28,981][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:53:29,310][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:53:29,639][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:53:29,971][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:53:30,292][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:53:30,619][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:53:30,945][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:53:31,274][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:53:31,598][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:53:31,925][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:53:32,252][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:53:32,582][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:53:32,921][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:53:33,247][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:53:33,574][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:53:33,901][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:53:34,230][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:53:34,561][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:53:34,894][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:53:35,222][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:53:35,549][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:53:35,876][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:53:36,204][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:53:36,532][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:53:36,864][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:53:37,190][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:53:37,517][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:53:37,844][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:53:38,597][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:53:39,371][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:53:39,372][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:53:39,374][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:53:41,405][__main__][INFO] - Iteration 141 took 22s (32.77% Gen, 58.34% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 13m 33s. Estimated total time: 19h 3m 25s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 6s, 500 more iterations: 3h 10m 34s.
[2025-11-13 08:53:41,408][__main__][INFO] - Starting iteration 141.
[2025-11-13 08:53:41,411][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:53:41,411][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:53:49,517][__main__][INFO] - Number of regex retries in iteration 141: 0
[2025-11-13 08:53:49,517][__main__][INFO] - agents played in iteration 141 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:53:49,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,078][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,118][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:53:50,119][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:53:50,119][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:53:50,919][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:53:51,216][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:53:51,543][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:53:51,873][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:53:52,200][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:53:52,529][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:53:52,856][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:53:53,182][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:53:53,511][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:53:53,846][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:53:54,173][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:53:54,503][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:53:54,829][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:53:55,169][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:53:55,495][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:53:55,820][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:53:56,147][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:53:56,478][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:53:56,806][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:53:57,132][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:53:57,458][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:53:57,786][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:53:58,112][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:53:58,439][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:53:58,764][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:53:59,092][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:53:59,418][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:53:59,745][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:54:00,078][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:54:00,402][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:54:00,728][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:54:01,057][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:54:01,384][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:54:02,155][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:54:02,930][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:02,931][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:02,933][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:03,949][__main__][INFO] - Iteration 142 took 22s (35.96% Gen, 59.52% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 56m 41s. Estimated total time: 18h 46m 56s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 49s.
[2025-11-13 08:54:03,951][__main__][INFO] - Starting iteration 142.
[2025-11-13 08:54:03,954][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:54:03,955][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:11,789][__main__][INFO] - Number of regex retries in iteration 142: 0
[2025-11-13 08:54:11,790][__main__][INFO] - agents played in iteration 142 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:54:12,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,295][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:12,376][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:12,377][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:13,154][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:54:13,450][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:54:13,777][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:54:14,102][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:54:14,428][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:54:14,761][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:54:15,087][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:54:15,419][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:54:15,749][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:54:16,076][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:54:16,404][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:54:16,729][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:54:17,057][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:54:17,384][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:54:17,711][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:54:18,038][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:54:18,366][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:54:18,692][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:54:19,018][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:54:19,343][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:54:19,670][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:54:19,999][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:54:20,325][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:54:20,652][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:54:20,979][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:54:21,306][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:54:21,631][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:54:21,958][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:54:22,284][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:54:22,610][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:54:22,937][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:54:23,263][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:54:23,591][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:54:24,361][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:54:25,117][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:25,118][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:25,120][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:26,105][__main__][INFO] - Iteration 143 took 22s (35.37% Gen, 60.17% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 37m 0s. Estimated total time: 18h 27m 37s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 55s, 500 more iterations: 3h 4m 36s.
[2025-11-13 08:54:26,107][__main__][INFO] - Starting iteration 143.
[2025-11-13 08:54:26,110][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:54:26,111][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:33,816][__main__][INFO] - Number of regex retries in iteration 143: 0
[2025-11-13 08:54:33,817][__main__][INFO] - agents played in iteration 143 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:54:34,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,400][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:34,401][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:34,402][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:35,173][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:54:35,471][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:54:35,799][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:54:36,126][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:54:36,453][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:54:36,781][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:54:37,111][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:54:37,440][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:54:37,767][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:54:38,092][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:54:38,419][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:54:38,746][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:54:39,074][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:54:39,400][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:54:39,727][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:54:40,053][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:54:40,385][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:54:40,706][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:54:41,036][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:54:41,363][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:54:41,694][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:54:42,017][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:54:42,343][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:54:42,671][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:54:43,000][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:54:43,328][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:54:43,655][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:54:43,982][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:54:44,309][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:54:44,638][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:54:44,969][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:54:45,295][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:54:45,623][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:54:46,377][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:54:47,138][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:54:47,139][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:54:47,141][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:54:48,116][__main__][INFO] - Iteration 144 took 22s (35.02% Gen, 60.54% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 29m 21s. Estimated total time: 18h 20m 21s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 40s, 500 more iterations: 3h 3m 23s.
[2025-11-13 08:54:48,118][__main__][INFO] - Starting iteration 144.
[2025-11-13 08:54:48,121][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:54:48,121][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:54:55,826][__main__][INFO] - Number of regex retries in iteration 144: 0
[2025-11-13 08:54:55,826][__main__][INFO] - agents played in iteration 144 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:54:56,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,351][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:54:56,432][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:54:56,432][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:54:57,209][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:54:57,506][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:54:57,835][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:54:58,162][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:54:58,488][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:54:58,815][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:54:59,141][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:54:59,467][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:54:59,795][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:55:00,121][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:55:00,449][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:55:00,775][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:55:01,102][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:55:01,429][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:55:01,756][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:55:02,083][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:55:02,409][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:55:02,736][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:55:03,063][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:55:03,392][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:55:03,720][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:55:04,046][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:55:04,372][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:55:04,699][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:55:05,028][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:55:05,355][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:55:05,689][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:55:06,018][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:55:06,347][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:55:06,676][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:55:07,003][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:55:07,336][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:55:07,662][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:55:08,437][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:55:09,240][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:09,242][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:09,246][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:10,499][__main__][INFO] - Iteration 145 took 22s (34.43% Gen, 59.97% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 47m 35s. Estimated total time: 18h 38m 56s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 17s, 500 more iterations: 3h 6m 29s.
[2025-11-13 08:55:10,501][__main__][INFO] - Starting iteration 145.
[2025-11-13 08:55:10,505][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:55:10,506][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:55:17,834][__main__][INFO] - Number of regex retries in iteration 145: 0
[2025-11-13 08:55:17,835][__main__][INFO] - agents played in iteration 145 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:55:18,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,342][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:18,425][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:55:18,425][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:55:19,201][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:55:19,499][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:55:19,827][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:55:20,156][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:55:20,488][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:55:20,814][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:55:21,141][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:55:21,480][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:55:21,808][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:55:22,134][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:55:22,461][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:55:22,792][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:55:23,119][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:55:23,446][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:55:23,772][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:55:24,101][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:55:24,429][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:55:24,756][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:55:25,084][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:55:25,410][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:55:25,737][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:55:26,067][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:55:26,399][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:55:26,721][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:55:27,048][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:55:27,379][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:55:27,706][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:55:28,033][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:55:28,360][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:55:28,688][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:55:29,014][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:55:29,341][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:55:29,668][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:55:30,424][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:55:31,185][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:31,186][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:31,188][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:32,279][__main__][INFO] - Iteration 146 took 21s (33.65% Gen, 61.32% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 17m 4s. Estimated total time: 18h 8m 48s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 17s, 500 more iterations: 3h 1m 28s.
[2025-11-13 08:55:32,281][__main__][INFO] - Starting iteration 146.
[2025-11-13 08:55:32,285][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:55:32,286][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:55:39,499][__main__][INFO] - Number of regex retries in iteration 146: 0
[2025-11-13 08:55:39,500][__main__][INFO] - agents played in iteration 146 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:55:39,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:55:40,085][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:55:40,086][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:55:40,863][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:55:41,173][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:55:41,505][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:55:41,833][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:55:42,160][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:55:42,492][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:55:42,818][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:55:43,146][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:55:43,476][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:55:43,798][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:55:44,124][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:55:44,451][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:55:44,781][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:55:45,106][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:55:45,435][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:55:45,761][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:55:46,088][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:55:46,415][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:55:46,741][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:55:47,069][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:55:47,398][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:55:47,720][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:55:48,046][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:55:48,374][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:55:48,700][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:55:49,027][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:55:49,353][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:55:49,681][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:55:50,008][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:55:50,334][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:55:50,661][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:55:50,987][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:55:51,314][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:55:52,013][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:55:52,767][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:55:52,769][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:55:52,771][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:55:53,875][__main__][INFO] - Iteration 147 took 21s (33.41% Gen, 61.46% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 7m 28s. Estimated total time: 17h 59m 33s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 59s, 500 more iterations: 2h 59m 55s.
[2025-11-13 08:55:53,878][__main__][INFO] - Starting iteration 147.
[2025-11-13 08:55:53,881][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:55:53,882][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:01,377][__main__][INFO] - Number of regex retries in iteration 147: 0
[2025-11-13 08:56:01,378][__main__][INFO] - agents played in iteration 147 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:56:01,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:01,853][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:01,905][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:01,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:01,946][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:01,946][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:02,727][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:56:03,025][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:56:03,354][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:56:03,681][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:56:04,009][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:56:04,336][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:56:04,662][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:56:04,989][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:56:05,315][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:56:05,644][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:56:05,975][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:56:06,301][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:56:06,627][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:56:06,953][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:56:07,280][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:56:07,607][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:56:07,934][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:56:08,259][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:56:08,587][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:56:08,914][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:56:09,240][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:56:09,566][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:56:09,893][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:56:10,218][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:56:10,545][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:56:10,871][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:56:11,198][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:56:11,525][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:56:11,851][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:56:12,178][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:56:12,505][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:56:12,831][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:56:13,157][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:56:13,833][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:56:14,578][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:14,580][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:14,581][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:56:15,603][__main__][INFO] - Iteration 148 took 21s (34.51% Gen, 60.78% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 13m 39s. Estimated total time: 18h 6m 6s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 12s, 500 more iterations: 3h 1m 1s.
[2025-11-13 08:56:15,605][__main__][INFO] - Starting iteration 148.
[2025-11-13 08:56:15,608][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:56:15,609][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:22,978][__main__][INFO] - Number of regex retries in iteration 148: 0
[2025-11-13 08:56:22,979][__main__][INFO] - agents played in iteration 148 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:56:23,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:23,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:23,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:23,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:23,549][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:23,549][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:24,280][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:56:24,578][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:56:24,906][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:56:25,242][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:56:25,576][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:56:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:56:26,231][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:56:26,558][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:56:26,888][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:56:27,215][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:56:27,542][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:56:27,869][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:56:28,199][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:56:28,528][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:56:28,857][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:56:29,187][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:56:29,517][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:56:29,844][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:56:30,171][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:56:30,499][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:56:30,832][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:56:31,158][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:56:31,485][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:56:31,813][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:56:32,139][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:56:32,469][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:56:32,798][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:56:33,125][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:56:33,454][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:56:33,780][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:56:34,108][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:56:34,440][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:56:34,776][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:56:35,480][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:56:36,250][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:36,252][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:36,253][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:56:37,267][__main__][INFO] - Iteration 149 took 21s (34.02% Gen, 61.29% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 10m 11s. Estimated total time: 18h 3m 0s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 6s, 500 more iterations: 3h 0m 30s.
[2025-11-13 08:56:37,269][__main__][INFO] - Starting iteration 149.
[2025-11-13 08:56:37,273][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:56:37,274][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:56:44,736][__main__][INFO] - Number of regex retries in iteration 149: 0
[2025-11-13 08:56:44,737][__main__][INFO] - agents played in iteration 149 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:56:45,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:45,248][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:45,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:45,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:56:45,329][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:56:45,330][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:56:46,039][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:56:46,337][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:56:46,667][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:56:47,001][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:56:47,331][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:56:47,661][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:56:47,991][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:56:48,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:56:48,648][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:56:48,975][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:56:49,309][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:56:49,642][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:56:49,972][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:56:50,301][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:56:50,628][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:56:50,956][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:56:51,286][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:56:51,619][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:56:51,945][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:56:52,275][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:56:52,603][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:56:52,932][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:56:53,268][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:56:53,596][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:56:53,922][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:56:54,248][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:56:54,582][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:56:54,908][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:56:55,234][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:56:55,560][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:56:55,900][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:56:56,227][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:56:56,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:56:57,253][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:56:57,994][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:56:57,996][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:56:57,997][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:56:59,120][__main__][INFO] - Iteration 150 took 21s (34.16% Gen, 60.69% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 19m 13s. Estimated total time: 18h 12m 24s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 24s, 500 more iterations: 3h 2m 4s.
[2025-11-13 08:56:59,122][__main__][INFO] - Starting iteration 150.
[2025-11-13 08:56:59,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 14 and human policies 1.
[2025-11-13 08:56:59,126][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:05,665][__main__][INFO] - Number of regex retries in iteration 150: 0
[2025-11-13 08:57:05,666][__main__][INFO] - agents played in iteration 150 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:57:06,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:06,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:06,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:06,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:06,254][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:06,254][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:07,003][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:57:07,302][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:57:07,629][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:57:07,956][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:57:08,282][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:57:08,609][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:57:08,942][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:57:09,273][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:57:09,601][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:57:09,936][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:57:10,263][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:57:10,592][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:57:10,918][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:57:11,249][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:57:11,576][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:57:11,911][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:57:12,237][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:57:12,564][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:57:12,894][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:57:13,221][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:57:13,548][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:57:13,875][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:57:14,201][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:57:14,528][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:57:14,854][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:57:15,181][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:57:15,509][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:57:15,836][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:57:16,163][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:57:16,490][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:57:16,817][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:57:17,143][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:57:17,470][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:57:18,146][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:57:18,878][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:57:18,880][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:57:18,881][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:57:21,389][__main__][INFO] - Iteration 151 took 22s (29.37% Gen, 59.35% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 39m 42s. Estimated total time: 18h 33m 15s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 6s, 500 more iterations: 3h 5m 32s.
[2025-11-13 08:57:21,391][__main__][INFO] - Starting iteration 151.
[2025-11-13 08:57:21,395][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:57:21,395][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:28,592][__main__][INFO] - Number of regex retries in iteration 151: 0
[2025-11-13 08:57:28,593][__main__][INFO] - agents played in iteration 151 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:57:29,039][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:29,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:29,121][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:29,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:29,513][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:29,513][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:30,258][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:57:30,557][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:57:30,885][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:57:31,215][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:57:31,549][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:57:31,879][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:57:32,212][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:57:32,540][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:57:32,875][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:57:33,203][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:57:33,537][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:57:33,863][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:57:34,191][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:57:34,523][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:57:34,852][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:57:35,178][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:57:35,511][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:57:35,837][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:57:36,162][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:57:36,491][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:57:36,820][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:57:37,154][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:57:37,482][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:57:37,814][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:57:38,143][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:57:38,473][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:57:38,805][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:57:39,135][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:57:39,467][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:57:39,797][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:57:40,126][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:57:40,464][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:57:40,798][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:57:41,486][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:57:42,220][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:57:42,222][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:57:42,224][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:57:43,226][__main__][INFO] - Iteration 152 took 21s (32.97% Gen, 62.43% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 17m 41s. Estimated total time: 18h 11m 35s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 23s, 500 more iterations: 3h 1m 55s.
[2025-11-13 08:57:43,228][__main__][INFO] - Starting iteration 152.
[2025-11-13 08:57:43,232][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:57:43,232][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:57:49,700][__main__][INFO] - Number of regex retries in iteration 152: 0
[2025-11-13 08:57:49,700][__main__][INFO] - agents played in iteration 152 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:57:50,154][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:50,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:50,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:50,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:57:50,275][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:57:50,275][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:57:51,037][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:57:51,334][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:57:51,662][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:57:51,987][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:57:52,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:57:52,644][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:57:52,971][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:57:53,299][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:57:53,625][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:57:53,953][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:57:54,280][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:57:54,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:57:54,939][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:57:55,269][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:57:55,597][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:57:55,928][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:57:56,250][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:57:56,579][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:57:56,906][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:57:57,234][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:57:57,559][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:57:57,886][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:57:58,211][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:57:58,538][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:57:58,864][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:57:59,191][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:57:59,518][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:57:59,844][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:58:00,173][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:58:00,500][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:58:00,826][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:58:01,152][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:58:01,481][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:58:02,164][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:58:02,909][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:02,911][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:02,912][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:03,912][__main__][INFO] - Iteration 153 took 20s (31.27% Gen, 63.88% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 19m 47s. Estimated total time: 17h 14m 2s. Time estimates for 10 more iterations: 3m 26s, 100 more iterations: 34m 28s, 500 more iterations: 2h 52m 20s.
[2025-11-13 08:58:03,914][__main__][INFO] - Starting iteration 153.
[2025-11-13 08:58:03,917][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:58:03,918][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:58:10,762][__main__][INFO] - Number of regex retries in iteration 153: 0
[2025-11-13 08:58:10,762][__main__][INFO] - agents played in iteration 153 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:58:11,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:11,263][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:11,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:11,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:11,344][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:58:11,344][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:58:12,104][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:58:12,402][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:58:12,728][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:58:13,061][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:58:13,389][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:58:13,717][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:58:14,044][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:58:14,371][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:58:14,704][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:58:15,034][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:58:15,360][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:58:15,688][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:58:16,021][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:58:16,357][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:58:16,692][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:58:17,020][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:58:17,349][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:58:17,677][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:58:18,004][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:58:18,334][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:58:18,663][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:58:18,990][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:58:19,317][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:58:19,644][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:58:19,971][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:58:20,298][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:58:20,631][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:58:20,961][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:58:21,291][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:58:21,625][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:58:21,952][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:58:22,285][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:58:22,615][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:58:23,304][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:58:24,056][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:24,057][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:24,059][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:25,061][__main__][INFO] - Iteration 154 took 21s (32.37% Gen, 62.88% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 42m 39s. Estimated total time: 17h 37m 15s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 14s, 500 more iterations: 2h 56m 12s.
[2025-11-13 08:58:25,063][__main__][INFO] - Starting iteration 154.
[2025-11-13 08:58:25,067][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:58:25,067][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:58:31,928][__main__][INFO] - Number of regex retries in iteration 154: 0
[2025-11-13 08:58:31,929][__main__][INFO] - agents played in iteration 154 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:58:32,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:32,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:32,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:32,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:32,503][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:58:32,504][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:58:33,269][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:58:33,567][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:58:33,894][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:58:34,221][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:58:34,548][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:58:34,875][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:58:35,208][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:58:35,541][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:58:35,872][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:58:36,199][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:58:36,531][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:58:36,869][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:58:37,196][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:58:37,525][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:58:37,852][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:58:38,183][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:58:38,513][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:58:38,843][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:58:39,178][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:58:39,510][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:58:39,837][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:58:40,163][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:58:40,490][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:58:40,822][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:58:41,149][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:58:41,479][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:58:41,809][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:58:42,133][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:58:42,460][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:58:42,787][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:58:43,119][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:58:43,440][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:58:43,768][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:58:44,453][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:58:45,200][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:58:45,201][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:58:45,203][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:58:46,199][__main__][INFO] - Iteration 155 took 21s (32.47% Gen, 62.81% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 41m 42s. Estimated total time: 17h 36m 39s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 13s, 500 more iterations: 2h 56m 6s.
[2025-11-13 08:58:46,201][__main__][INFO] - Starting iteration 155.
[2025-11-13 08:58:46,204][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:58:46,205][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:58:52,832][__main__][INFO] - Number of regex retries in iteration 155: 0
[2025-11-13 08:58:52,833][__main__][INFO] - agents played in iteration 155 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:58:53,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:53,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:53,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:53,410][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:58:53,410][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:58:53,410][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:58:54,174][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:58:54,472][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:58:54,799][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:58:55,125][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:58:55,453][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:58:55,781][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:58:56,109][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:58:56,437][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:58:56,765][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:58:57,097][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:58:57,424][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:58:57,754][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:58:58,088][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:58:58,415][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:58:58,744][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:58:59,074][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:58:59,400][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:58:59,727][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:59:00,057][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:59:00,385][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:59:00,713][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:59:01,043][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:59:01,371][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:59:01,701][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:59:02,028][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:59:02,356][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:59:02,690][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:59:03,020][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:59:03,348][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:59:03,675][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:59:04,002][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:59:04,332][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:59:04,666][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:59:05,356][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:59:06,112][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:06,113][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:06,115][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:07,095][__main__][INFO] - Iteration 156 took 20s (31.72% Gen, 63.57% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 29m 18s. Estimated total time: 17h 24m 36s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 49s, 500 more iterations: 2h 54m 6s.
[2025-11-13 08:59:07,098][__main__][INFO] - Starting iteration 156.
[2025-11-13 08:59:07,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:59:07,101][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:13,585][__main__][INFO] - Number of regex retries in iteration 156: 0
[2025-11-13 08:59:13,586][__main__][INFO] - agents played in iteration 156 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:59:14,057][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:14,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:14,137][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:14,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:14,178][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:14,178][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:14,966][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:59:15,263][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:59:15,591][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:59:15,918][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:59:16,245][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:59:16,573][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:59:16,900][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:59:17,229][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:59:17,556][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:59:17,883][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:59:18,212][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:59:18,540][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:59:18,874][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:59:19,204][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:59:19,534][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:59:19,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:59:20,192][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:59:20,520][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:59:20,851][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:59:21,179][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:59:21,512][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:59:21,846][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:59:22,173][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:59:22,506][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:59:22,837][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:59:23,166][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:59:23,496][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:59:23,824][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:59:24,151][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:59:24,483][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:59:24,810][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:59:25,141][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:59:25,464][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:59:26,153][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:59:26,888][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:26,889][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:26,891][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:27,902][__main__][INFO] - Iteration 157 took 20s (31.17% Gen, 63.96% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 24m 26s. Estimated total time: 17h 20m 5s. Time estimates for 10 more iterations: 3m 28s, 100 more iterations: 34m 40s, 500 more iterations: 2h 53m 20s.
[2025-11-13 08:59:27,904][__main__][INFO] - Starting iteration 157.
[2025-11-13 08:59:27,907][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:59:27,908][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:34,375][__main__][INFO] - Number of regex retries in iteration 157: 0
[2025-11-13 08:59:34,375][__main__][INFO] - agents played in iteration 157 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:59:34,851][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:34,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:34,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:34,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:34,975][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:34,976][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:35,836][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:59:36,135][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:59:36,463][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:59:36,791][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:59:37,127][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:59:37,447][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:59:37,774][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:59:38,102][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:59:38,437][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:59:38,758][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:59:39,085][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 08:59:39,413][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 08:59:39,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 08:59:40,068][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 08:59:40,396][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 08:59:40,722][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 08:59:41,051][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 08:59:41,379][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 08:59:41,708][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 08:59:42,036][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 08:59:42,366][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 08:59:42,697][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 08:59:43,026][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 08:59:43,354][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 08:59:43,683][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 08:59:44,014][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 08:59:44,342][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 08:59:44,670][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 08:59:45,001][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 08:59:45,329][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 08:59:45,656][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 08:59:45,982][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 08:59:46,311][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 08:59:46,986][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 08:59:47,734][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 08:59:47,735][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 08:59:47,737][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 08:59:48,877][__main__][INFO] - Iteration 158 took 20s (30.84% Gen, 63.72% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 32m 32s. Estimated total time: 17h 28m 32s. Time estimates for 10 more iterations: 3m 29s, 100 more iterations: 34m 57s, 500 more iterations: 2h 54m 45s.
[2025-11-13 08:59:48,879][__main__][INFO] - Starting iteration 158.
[2025-11-13 08:59:48,883][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 08:59:48,883][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 08:59:55,360][__main__][INFO] - Number of regex retries in iteration 158: 0
[2025-11-13 08:59:55,361][__main__][INFO] - agents played in iteration 158 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 08:59:55,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:55,847][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:55,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:55,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 08:59:55,928][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 08:59:55,929][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 08:59:56,720][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 08:59:57,018][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 08:59:57,346][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 08:59:57,675][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 08:59:58,003][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 08:59:58,332][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 08:59:58,658][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 08:59:58,988][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 08:59:59,322][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 08:59:59,643][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 08:59:59,971][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:00:00,300][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:00:00,633][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:00:00,954][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:00:01,281][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:00:01,609][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:00:01,940][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:00:02,265][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:00:02,592][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:00:02,922][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:00:03,249][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:00:03,578][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:00:03,906][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:00:04,236][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:00:04,566][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:00:04,900][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:00:05,230][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:00:05,558][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:00:05,889][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:00:06,228][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:00:06,558][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:00:06,886][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:00:07,218][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:00:07,930][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:00:08,689][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:00:08,690][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:00:08,692][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:00:09,949][__main__][INFO] - Iteration 159 took 21s (30.75% Gen, 63.28% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 37m 0s. Estimated total time: 17h 33m 21s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 6s, 500 more iterations: 2h 55m 33s.
[2025-11-13 09:00:09,951][__main__][INFO] - Starting iteration 159.
[2025-11-13 09:00:09,955][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 09:00:09,955][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:00:16,376][__main__][INFO] - Number of regex retries in iteration 159: 0
[2025-11-13 09:00:16,377][__main__][INFO] - agents played in iteration 159 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:00:16,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:16,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:16,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:16,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:16,935][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:00:16,935][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:00:17,636][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:00:17,934][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:00:18,270][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:00:18,602][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:00:18,931][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:00:19,266][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:00:19,599][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:00:19,929][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:00:20,260][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:00:20,588][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:00:20,916][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:00:21,244][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:00:21,572][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:00:21,903][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:00:22,231][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:00:22,559][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:00:22,887][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:00:23,217][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:00:23,544][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:00:23,872][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:00:24,201][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:00:24,533][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:00:24,859][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:00:25,188][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:00:25,516][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:00:25,844][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:00:26,171][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:00:26,497][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:00:26,826][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:00:27,156][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:00:27,486][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:00:27,816][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:00:28,144][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:00:28,895][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:00:29,656][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:00:29,657][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:00:29,659][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:00:30,678][__main__][INFO] - Iteration 160 took 20s (30.98% Gen, 64.09% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 19m 30s. Estimated total time: 17h 16m 12s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 32s, 500 more iterations: 2h 52m 42s.
[2025-11-13 09:00:30,680][__main__][INFO] - Starting iteration 160.
[2025-11-13 09:00:30,683][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 15 and human policies 1.
[2025-11-13 09:00:30,684][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:00:37,507][__main__][INFO] - Number of regex retries in iteration 160: 0
[2025-11-13 09:00:37,508][__main__][INFO] - agents played in iteration 160 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:00:37,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:37,984][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:38,023][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:38,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:00:38,063][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:00:38,064][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:00:38,769][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:00:39,066][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:00:39,397][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:00:39,725][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:00:40,056][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:00:40,386][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:00:40,719][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:00:41,054][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:00:41,382][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:00:41,711][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:00:42,040][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:00:42,368][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:00:42,696][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:00:43,024][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:00:43,352][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:00:43,680][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:00:44,007][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:00:44,338][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:00:44,665][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:00:44,992][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:00:45,325][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:00:45,646][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:00:45,973][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:00:46,301][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:00:46,629][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:00:46,952][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:00:47,285][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:00:47,615][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:00:47,947][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:00:48,270][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:00:48,605][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:00:48,932][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:00:49,263][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:00:50,053][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:00:50,828][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:00:50,830][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:00:50,832][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:00:52,794][__main__][INFO] - Iteration 161 took 22s (30.86% Gen, 60.26% Train). Generation: 6s, Training: 13s. Estimated remaining time: 17h 28m 31s. Estimated total time: 18h 25m 35s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 51s, 500 more iterations: 3h 4m 15s.
[2025-11-13 09:00:52,796][__main__][INFO] - Starting iteration 161.
[2025-11-13 09:00:52,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:00:52,800][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:00,120][__main__][INFO] - Number of regex retries in iteration 161: 0
[2025-11-13 09:01:00,121][__main__][INFO] - agents played in iteration 161 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:01:00,556][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:00,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:00,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:00,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:00,676][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:00,676][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:01,377][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:01:01,675][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:01:02,002][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:01:02,332][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:01:02,659][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:01:02,986][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:01:03,314][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:01:03,643][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:01:03,977][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:01:04,306][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:01:04,637][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:01:04,965][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:01:05,293][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:01:05,620][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:01:05,948][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:01:06,276][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:01:06,603][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:01:06,930][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:01:07,257][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:01:07,586][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:01:07,914][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:01:08,242][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:01:08,569][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:01:08,898][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:01:09,225][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:01:09,553][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:01:09,885][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:01:10,214][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:01:10,548][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:01:10,877][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:01:11,205][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:01:11,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:01:11,862][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:01:12,582][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:01:13,349][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:13,350][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:13,352][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:14,526][__main__][INFO] - Iteration 162 took 21s (33.70% Gen, 60.89% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 8m 58s. Estimated total time: 18h 6m 24s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 12s, 500 more iterations: 3h 1m 4s.
[2025-11-13 09:01:14,529][__main__][INFO] - Starting iteration 162.
[2025-11-13 09:01:14,532][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:01:14,533][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:20,969][__main__][INFO] - Number of regex retries in iteration 162: 0
[2025-11-13 09:01:20,970][__main__][INFO] - agents played in iteration 162 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:01:21,417][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:21,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:21,498][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:21,537][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:21,537][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:21,538][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:22,236][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:01:22,535][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:01:22,864][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:01:23,192][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:01:23,521][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:01:23,853][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:01:24,179][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:01:24,505][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:01:24,831][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:01:25,162][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:01:25,488][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:01:25,818][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:01:26,148][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:01:26,478][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:01:26,813][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:01:27,147][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:01:27,477][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:01:27,806][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:01:28,136][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:01:28,463][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:01:28,789][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:01:29,116][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:01:29,444][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:01:29,771][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:01:30,097][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:01:30,424][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:01:30,758][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:01:31,085][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:01:31,411][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:01:31,738][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:01:32,065][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:01:32,393][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:01:32,721][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:01:33,470][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:01:34,225][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:34,227][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:34,228][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:35,243][__main__][INFO] - Iteration 163 took 20s (31.08% Gen, 64.01% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 17m 49s. Estimated total time: 17h 15m 36s. Time estimates for 10 more iterations: 3m 27s, 100 more iterations: 34m 31s, 500 more iterations: 2h 52m 36s.
[2025-11-13 09:01:35,245][__main__][INFO] - Starting iteration 163.
[2025-11-13 09:01:35,249][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:01:35,249][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:01:42,172][__main__][INFO] - Number of regex retries in iteration 163: 0
[2025-11-13 09:01:42,173][__main__][INFO] - agents played in iteration 163 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:01:42,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:42,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:42,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:42,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:01:42,750][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:01:42,751][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:01:43,443][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:01:43,740][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:01:44,067][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:01:44,392][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:01:44,720][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:01:45,046][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:01:45,371][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:01:45,697][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:01:46,025][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:01:46,352][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:01:46,678][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:01:47,004][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:01:47,333][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:01:47,664][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:01:47,992][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:01:48,324][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:01:48,650][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:01:48,980][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:01:49,308][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:01:49,636][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:01:49,969][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:01:50,296][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:01:50,623][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:01:50,950][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:01:51,278][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:01:51,605][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:01:51,932][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:01:52,259][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:01:52,587][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:01:52,915][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:01:53,247][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:01:53,574][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:01:53,903][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:01:54,646][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:01:55,380][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:01:55,382][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:01:55,384][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:01:56,384][__main__][INFO] - Iteration 164 took 21s (32.76% Gen, 62.50% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 38m 40s. Estimated total time: 17h 36m 47s. Time estimates for 10 more iterations: 3m 31s, 100 more iterations: 35m 13s, 500 more iterations: 2h 56m 7s.
[2025-11-13 09:01:56,386][__main__][INFO] - Starting iteration 164.
[2025-11-13 09:01:56,389][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:01:56,389][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:02:03,316][__main__][INFO] - Number of regex retries in iteration 164: 0
[2025-11-13 09:02:03,317][__main__][INFO] - agents played in iteration 164 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:02:03,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:03,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:03,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:03,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:03,894][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:02:03,894][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:02:04,586][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:02:04,881][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:02:05,206][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:02:05,532][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:02:05,858][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:02:06,189][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:02:06,514][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:02:06,840][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:02:07,166][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:02:07,492][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:02:07,817][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:02:08,144][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:02:08,469][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:02:08,797][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:02:09,122][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:02:09,450][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:02:09,778][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:02:10,106][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:02:10,439][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:02:10,768][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:02:11,097][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:02:11,425][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:02:11,752][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:02:12,079][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:02:12,405][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:02:12,732][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:02:13,073][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:02:13,401][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:02:13,728][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:02:14,054][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:02:14,381][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:02:14,708][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:02:15,035][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:02:15,784][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:02:16,548][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:16,550][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:16,551][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:02:17,838][__main__][INFO] - Iteration 165 took 21s (32.30% Gen, 61.70% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 53m 59s. Estimated total time: 17h 52m 28s. Time estimates for 10 more iterations: 3m 34s, 100 more iterations: 35m 44s, 500 more iterations: 2h 58m 44s.
[2025-11-13 09:02:17,840][__main__][INFO] - Starting iteration 165.
[2025-11-13 09:02:17,843][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:02:17,844][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:02:24,823][__main__][INFO] - Number of regex retries in iteration 165: 0
[2025-11-13 09:02:24,823][__main__][INFO] - agents played in iteration 165 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:02:25,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:25,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:25,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:25,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:25,387][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:02:25,387][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:02:26,115][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:02:26,411][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:02:26,738][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:02:27,064][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:02:27,395][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:02:27,730][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:02:28,062][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:02:28,392][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:02:28,721][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:02:29,051][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:02:29,377][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:02:29,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:02:30,038][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:02:30,367][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:02:30,694][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:02:31,026][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:02:31,360][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:02:31,686][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:02:32,017][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:02:32,350][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:02:32,681][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:02:33,012][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:02:33,339][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:02:33,667][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:02:34,001][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:02:34,328][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:02:34,654][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:02:34,981][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:02:35,308][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:02:35,635][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:02:35,962][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:02:36,289][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:02:36,624][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:02:37,357][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:02:38,115][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:38,117][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:38,119][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:02:39,135][__main__][INFO] - Iteration 166 took 21s (32.78% Gen, 62.44% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 45m 46s. Estimated total time: 17h 44m 37s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 29s, 500 more iterations: 2h 57m 26s.
[2025-11-13 09:02:39,137][__main__][INFO] - Starting iteration 166.
[2025-11-13 09:02:39,141][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:02:39,141][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:02:45,976][__main__][INFO] - Number of regex retries in iteration 166: 0
[2025-11-13 09:02:45,977][__main__][INFO] - agents played in iteration 166 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:02:46,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:46,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:46,493][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:46,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:02:46,533][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:02:46,533][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:02:47,251][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:02:47,547][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:02:47,874][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:02:48,203][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:02:48,541][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:02:48,868][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:02:49,195][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:02:49,524][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:02:49,865][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:02:50,192][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:02:50,519][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:02:50,847][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:02:51,175][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:02:51,504][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:02:51,832][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:02:52,159][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:02:52,491][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:02:52,811][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:02:53,137][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:02:53,464][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:02:53,797][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:02:54,120][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:02:54,449][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:02:54,777][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:02:55,107][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:02:55,439][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:02:55,770][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:02:56,105][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:02:56,432][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:02:56,770][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:02:57,096][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:02:57,423][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:02:57,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:02:58,506][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:02:59,260][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:02:59,261][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:02:59,263][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:03:00,188][__main__][INFO] - Iteration 167 took 21s (32.47% Gen, 63.12% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 33m 16s. Estimated total time: 17h 32m 27s. Time estimates for 10 more iterations: 3m 30s, 100 more iterations: 35m 4s, 500 more iterations: 2h 55m 24s.
[2025-11-13 09:03:00,191][__main__][INFO] - Starting iteration 167.
[2025-11-13 09:03:00,194][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:03:00,194][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:07,428][__main__][INFO] - Number of regex retries in iteration 167: 0
[2025-11-13 09:03:07,428][__main__][INFO] - agents played in iteration 167 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:03:07,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:07,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:07,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:07,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:07,995][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:07,995][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:08,707][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:03:09,005][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:03:09,331][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:03:09,659][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:03:09,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:03:10,316][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:03:10,643][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:03:10,969][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:03:11,296][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:03:11,622][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:03:11,947][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:03:12,274][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:03:12,601][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:03:12,928][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:03:13,255][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:03:13,589][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:03:13,911][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:03:14,237][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:03:14,565][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:03:14,891][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:03:15,217][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:03:15,545][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:03:15,875][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:03:16,205][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:03:16,531][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:03:16,863][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:03:17,192][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:03:17,519][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:03:17,856][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:03:18,184][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:03:18,511][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:03:18,839][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:03:19,180][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:03:19,911][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:03:20,658][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:03:20,659][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:03:20,661][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:03:21,645][__main__][INFO] - Iteration 168 took 21s (33.72% Gen, 61.68% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 53m 4s. Estimated total time: 17h 52m 36s. Time estimates for 10 more iterations: 3m 34s, 100 more iterations: 35m 45s, 500 more iterations: 2h 58m 46s.
[2025-11-13 09:03:21,647][__main__][INFO] - Starting iteration 168.
[2025-11-13 09:03:21,649][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:03:21,650][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:28,987][__main__][INFO] - Number of regex retries in iteration 168: 0
[2025-11-13 09:03:28,987][__main__][INFO] - agents played in iteration 168 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:03:29,439][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:29,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:29,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:29,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:29,557][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:29,558][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:30,262][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:03:30,559][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:03:30,894][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:03:31,220][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:03:31,549][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:03:31,876][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:03:32,205][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:03:32,532][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:03:32,858][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:03:33,196][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:03:33,521][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:03:33,848][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:03:34,174][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:03:34,500][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:03:34,827][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:03:35,153][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:03:35,478][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:03:35,805][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:03:36,133][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:03:36,460][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:03:36,787][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:03:37,115][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:03:37,441][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:03:37,769][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:03:38,096][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:03:38,425][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:03:38,759][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:03:39,088][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:03:39,420][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:03:39,754][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:03:40,090][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:03:40,417][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:03:40,745][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:03:41,511][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:03:42,223][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:03:42,225][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:03:42,227][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:03:43,157][__main__][INFO] - Iteration 169 took 21s (34.11% Gen, 61.56% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 55m 30s. Estimated total time: 17h 55m 25s. Time estimates for 10 more iterations: 3m 35s, 100 more iterations: 35m 50s, 500 more iterations: 2h 59m 14s.
[2025-11-13 09:03:43,159][__main__][INFO] - Starting iteration 169.
[2025-11-13 09:03:43,161][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:03:43,162][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:03:50,256][__main__][INFO] - Number of regex retries in iteration 169: 0
[2025-11-13 09:03:50,256][__main__][INFO] - agents played in iteration 169 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:03:50,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:50,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:50,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:50,814][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:03:50,815][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:03:50,815][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:03:51,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:03:51,819][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:03:52,145][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:03:52,470][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:03:52,798][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:03:53,124][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:03:53,450][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:03:53,775][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:03:54,103][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:03:54,433][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:03:54,759][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:03:55,087][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:03:55,413][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:03:55,739][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:03:56,067][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:03:56,394][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:03:56,720][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:03:57,046][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:03:57,382][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:03:57,708][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:03:58,034][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:03:58,360][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:03:58,700][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:03:59,031][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:03:59,357][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:03:59,687][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:04:00,021][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:04:00,353][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:04:00,682][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:04:01,014][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:04:01,346][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:04:01,679][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:04:02,007][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:04:02,777][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:04:03,488][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:03,489][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:03,491][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:04,440][__main__][INFO] - Iteration 170 took 21s (33.34% Gen, 62.19% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 43m 42s. Estimated total time: 17h 43m 57s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 27s, 500 more iterations: 2h 57m 19s.
[2025-11-13 09:04:04,442][__main__][INFO] - Starting iteration 170.
[2025-11-13 09:04:04,445][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 16 and human policies 1.
[2025-11-13 09:04:04,445][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:12,019][__main__][INFO] - Number of regex retries in iteration 170: 0
[2025-11-13 09:04:12,020][__main__][INFO] - agents played in iteration 170 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:04:12,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:12,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:12,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:12,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:12,573][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:12,573][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:13,295][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:04:13,592][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:04:13,920][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:04:14,246][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:04:14,573][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:04:14,898][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:04:15,225][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:04:15,551][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:04:15,877][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:04:16,208][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:04:16,534][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:04:16,861][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:04:17,188][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:04:17,517][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:04:17,844][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:04:18,171][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:04:18,497][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:04:18,824][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:04:19,151][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:04:19,478][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:04:19,804][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:04:20,132][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:04:20,459][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:04:20,789][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:04:21,115][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:04:21,443][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:04:21,769][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:04:22,098][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:04:22,426][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:04:22,760][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:04:23,090][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:04:23,420][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:04:23,748][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:04:24,510][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:04:25,228][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:25,230][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:25,231][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:27,038][__main__][INFO] - Iteration 171 took 22s (33.52% Gen, 58.47% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 49m 6s. Estimated total time: 18h 49m 44s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 39s, 500 more iterations: 3h 8m 17s.
[2025-11-13 09:04:27,041][__main__][INFO] - Starting iteration 171.
[2025-11-13 09:04:27,044][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:04:27,044][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:34,954][__main__][INFO] - Number of regex retries in iteration 171: 0
[2025-11-13 09:04:34,955][__main__][INFO] - agents played in iteration 171 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:04:35,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:35,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:35,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:35,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:35,510][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:35,510][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:36,221][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:04:36,517][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:04:36,843][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:04:37,168][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:04:37,494][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:04:37,819][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:04:38,146][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:04:38,471][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:04:38,797][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:04:39,125][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:04:39,452][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:04:39,783][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:04:40,109][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:04:40,436][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:04:40,761][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:04:41,088][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:04:41,414][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:04:41,741][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:04:42,066][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:04:42,393][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:04:42,718][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:04:43,044][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:04:43,370][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:04:43,701][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:04:44,029][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:04:44,364][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:04:44,692][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:04:45,019][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:04:45,348][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:04:45,683][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:04:46,017][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:04:46,347][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:04:46,681][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:04:47,427][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:04:48,146][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:04:48,147][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:04:48,149][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:04:49,075][__main__][INFO] - Iteration 172 took 22s (35.90% Gen, 59.88% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 20m 38s. Estimated total time: 18h 21m 38s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 43s, 500 more iterations: 3h 3m 36s.
[2025-11-13 09:04:49,077][__main__][INFO] - Starting iteration 172.
[2025-11-13 09:04:49,080][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:04:49,081][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:04:57,181][__main__][INFO] - Number of regex retries in iteration 172: 0
[2025-11-13 09:04:57,182][__main__][INFO] - agents played in iteration 172 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:04:57,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:57,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:57,713][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:57,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:04:57,753][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:04:57,754][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:04:58,502][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:04:58,798][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:04:59,129][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:04:59,458][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:04:59,786][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:05:00,119][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:05:00,443][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:05:00,771][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:05:01,100][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:05:01,431][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:05:01,758][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:05:02,087][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:05:02,413][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:05:02,741][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:05:03,069][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:05:03,402][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:05:03,733][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:05:04,066][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:05:04,396][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:05:04,728][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:05:05,056][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:05:05,382][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:05:05,711][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:05:06,040][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:05:06,372][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:05:06,703][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:05:07,030][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:05:07,356][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:05:07,684][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:05:08,012][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:05:08,340][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:05:08,669][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:05:08,996][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:05:09,764][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:05:10,491][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:10,492][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:10,494][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:11,419][__main__][INFO] - Iteration 173 took 22s (36.26% Gen, 59.59% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 35m 37s. Estimated total time: 18h 37m 0s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 14s, 500 more iterations: 3h 6m 10s.
[2025-11-13 09:05:11,421][__main__][INFO] - Starting iteration 173.
[2025-11-13 09:05:11,424][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:05:11,425][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:05:19,454][__main__][INFO] - Number of regex retries in iteration 173: 0
[2025-11-13 09:05:19,455][__main__][INFO] - agents played in iteration 173 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:05:19,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:19,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:20,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:20,042][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:20,043][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:05:20,043][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:05:20,790][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:05:21,089][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:05:21,417][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:05:21,744][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:05:22,069][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:05:22,395][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:05:22,723][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:05:23,050][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:05:23,376][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:05:23,702][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:05:24,028][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:05:24,354][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:05:24,680][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:05:25,009][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:05:25,333][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:05:25,660][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:05:25,987][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:05:26,313][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:05:26,640][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:05:26,966][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:05:27,293][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:05:27,623][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:05:27,945][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:05:28,272][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:05:28,600][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:05:28,927][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:05:29,253][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:05:29,580][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:05:29,907][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:05:30,233][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:05:30,562][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:05:30,889][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:05:31,217][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:05:31,969][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:05:32,721][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:32,723][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:32,724][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:33,686][__main__][INFO] - Iteration 174 took 22s (36.07% Gen, 59.61% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 31m 22s. Estimated total time: 18h 33m 7s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 6s, 500 more iterations: 3h 5m 31s.
[2025-11-13 09:05:33,688][__main__][INFO] - Starting iteration 174.
[2025-11-13 09:05:33,691][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:05:33,692][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:05:41,930][__main__][INFO] - Number of regex retries in iteration 174: 0
[2025-11-13 09:05:41,931][__main__][INFO] - agents played in iteration 174 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:05:42,404][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:05:42,527][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:05:42,527][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:05:43,278][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:05:43,576][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:05:43,905][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:05:44,237][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:05:44,562][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:05:44,887][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:05:45,212][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:05:45,539][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:05:45,871][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:05:46,201][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:05:46,527][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:05:46,857][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:05:47,186][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:05:47,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:05:47,850][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:05:48,183][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:05:48,509][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:05:48,841][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:05:49,172][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:05:49,506][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:05:49,839][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:05:50,166][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:05:50,492][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:05:50,823][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:05:51,150][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:05:51,477][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:05:51,803][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:05:52,129][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:05:52,455][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:05:52,787][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:05:53,116][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:05:53,444][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:05:53,772][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:05:54,529][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:05:55,292][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:05:55,293][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:05:55,295][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:05:56,227][__main__][INFO] - Iteration 175 took 22s (36.56% Gen, 59.30% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 44m 43s. Estimated total time: 18h 46m 51s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 33s, 500 more iterations: 3h 7m 48s.
[2025-11-13 09:05:56,229][__main__][INFO] - Starting iteration 175.
[2025-11-13 09:05:56,232][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:05:56,233][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:04,304][__main__][INFO] - Number of regex retries in iteration 175: 0
[2025-11-13 09:06:04,305][__main__][INFO] - agents played in iteration 175 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:06:04,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:04,908][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:04,909][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:05,684][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:06:05,988][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:06:06,316][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:06:06,644][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:06:06,970][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:06:07,305][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:06:07,631][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:06:07,959][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:06:08,286][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:06:08,613][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:06:08,941][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:06:09,268][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:06:09,594][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:06:09,924][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:06:10,250][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:06:10,576][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:06:10,903][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:06:11,229][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:06:11,555][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:06:11,882][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:06:12,210][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:06:12,537][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:06:12,863][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:06:13,190][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:06:13,517][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:06:13,844][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:06:14,171][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:06:14,498][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:06:14,825][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:06:15,151][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:06:15,477][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:06:15,805][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:06:16,133][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:06:16,915][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:06:17,671][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:06:17,673][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:06:17,675][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:06:18,698][__main__][INFO] - Iteration 176 took 22s (35.93% Gen, 59.51% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 40m 51s. Estimated total time: 18h 43m 21s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 26s, 500 more iterations: 3h 7m 13s.
[2025-11-13 09:06:18,701][__main__][INFO] - Starting iteration 176.
[2025-11-13 09:06:18,704][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:06:18,704][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:27,258][__main__][INFO] - Number of regex retries in iteration 176: 0
[2025-11-13 09:06:27,259][__main__][INFO] - agents played in iteration 176 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:06:27,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:27,845][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:27,845][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:28,628][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:06:28,925][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:06:29,251][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:06:29,581][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:06:29,906][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:06:30,232][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:06:30,558][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:06:30,886][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:06:31,215][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:06:31,544][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:06:31,870][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:06:32,200][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:06:32,526][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:06:32,853][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:06:33,178][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:06:33,512][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:06:33,838][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:06:34,166][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:06:34,493][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:06:34,832][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:06:35,159][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:06:35,489][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:06:35,819][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:06:36,150][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:06:36,477][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:06:36,806][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:06:37,136][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:06:37,459][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:06:37,786][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:06:38,113][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:06:38,450][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:06:38,776][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:06:39,105][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:06:39,881][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:06:40,631][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:06:40,632][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:06:40,634][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:06:41,912][__main__][INFO] - Iteration 177 took 23s (36.86% Gen, 57.63% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 17m 34s. Estimated total time: 19h 20m 27s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 40s, 500 more iterations: 3h 13m 24s.
[2025-11-13 09:06:41,914][__main__][INFO] - Starting iteration 177.
[2025-11-13 09:06:41,916][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:06:41,917][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:06:50,301][__main__][INFO] - Number of regex retries in iteration 177: 0
[2025-11-13 09:06:50,302][__main__][INFO] - agents played in iteration 177 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:06:50,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,847][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,888][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:06:50,889][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:06:50,889][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:06:51,675][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:06:51,971][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:06:52,299][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:06:52,629][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:06:52,959][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:06:53,283][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:06:53,610][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:06:53,936][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:06:54,265][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:06:54,589][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:06:54,915][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:06:55,243][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:06:55,570][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:06:55,897][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:06:56,227][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:06:56,557][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:06:56,883][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:06:57,216][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:06:57,545][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:06:57,874][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:06:58,204][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:06:58,536][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:06:58,865][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:06:59,191][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:06:59,520][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:06:59,854][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:07:00,184][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:07:00,513][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:07:00,843][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:07:01,168][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:07:01,495][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:07:01,825][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:07:02,160][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:07:02,956][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:07:03,720][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:03,722][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:03,723][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:04,672][__main__][INFO] - Iteration 178 took 22s (36.84% Gen, 58.98% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 54m 33s. Estimated total time: 18h 57m 49s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 55s, 500 more iterations: 3h 9m 38s.
[2025-11-13 09:07:04,674][__main__][INFO] - Starting iteration 178.
[2025-11-13 09:07:04,676][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:07:04,677][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:07:12,972][__main__][INFO] - Number of regex retries in iteration 178: 0
[2025-11-13 09:07:12,972][__main__][INFO] - agents played in iteration 178 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:07:13,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,519][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,560][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:13,561][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:07:13,561][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:07:14,348][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:07:14,644][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:07:14,970][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:07:15,297][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:07:15,624][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:07:15,950][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:07:16,277][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:07:16,604][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:07:16,937][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:07:17,263][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:07:17,591][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:07:17,917][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:07:18,252][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:07:18,578][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:07:18,907][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:07:19,233][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:07:19,563][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:07:19,890][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:07:20,219][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:07:20,551][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:07:20,872][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:07:21,199][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:07:21,529][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:07:21,864][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:07:22,185][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:07:22,511][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:07:22,836][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:07:23,163][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:07:23,490][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:07:23,816][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:07:24,145][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:07:24,471][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:07:24,798][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:07:25,564][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:07:26,320][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:26,322][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:26,324][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:27,587][__main__][INFO] - Iteration 179 took 22s (36.21% Gen, 58.28% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 1m 55s. Estimated total time: 19h 5m 33s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 11s, 500 more iterations: 3h 10m 55s.
[2025-11-13 09:07:27,589][__main__][INFO] - Starting iteration 179.
[2025-11-13 09:07:27,591][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:07:27,592][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:07:36,056][__main__][INFO] - Number of regex retries in iteration 179: 0
[2025-11-13 09:07:36,057][__main__][INFO] - agents played in iteration 179 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:07:36,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,570][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,651][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:36,652][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:07:36,652][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:07:37,436][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:07:37,734][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:07:38,061][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:07:38,391][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:07:38,718][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:07:39,045][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:07:39,374][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:07:39,701][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:07:40,027][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:07:40,360][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:07:40,681][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:07:41,008][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:07:41,337][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:07:41,668][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:07:41,991][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:07:42,318][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:07:42,645][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:07:42,974][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:07:43,300][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:07:43,627][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:07:43,953][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:07:44,281][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:07:44,606][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:07:44,932][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:07:45,260][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:07:45,585][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:07:45,911][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:07:46,237][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:07:46,565][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:07:46,891][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:07:47,220][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:07:47,548][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:07:47,877][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:07:48,630][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:07:49,383][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:07:49,385][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:07:49,386][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:07:50,388][__main__][INFO] - Iteration 180 took 22s (37.13% Gen, 58.47% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 55m 51s. Estimated total time: 18h 59m 53s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 59s, 500 more iterations: 3h 9m 58s.
[2025-11-13 09:07:50,391][__main__][INFO] - Starting iteration 180.
[2025-11-13 09:07:50,394][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 17 and human policies 1.
[2025-11-13 09:07:50,394][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:07:58,864][__main__][INFO] - Number of regex retries in iteration 180: 0
[2025-11-13 09:07:58,865][__main__][INFO] - agents played in iteration 180 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:07:59,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,362][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,443][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:07:59,443][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:07:59,444][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:00,224][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:08:00,522][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:08:00,849][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:08:01,178][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:08:01,505][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:08:01,832][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:08:02,159][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:08:02,486][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:08:02,813][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:08:03,139][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:08:03,467][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:08:03,794][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:08:04,119][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:08:04,445][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:08:04,777][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:08:05,099][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:08:05,425][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:08:05,752][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:08:06,085][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:08:06,414][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:08:06,743][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:08:07,069][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:08:07,397][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:08:07,724][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:08:08,051][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:08:08,377][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:08:08,706][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:08:09,032][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:08:09,358][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:08:09,684][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:08:10,010][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:08:10,337][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:08:10,667][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:08:11,431][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:08:12,181][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:12,183][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:12,184][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:08:14,099][__main__][INFO] - Iteration 181 took 23s (35.73% Gen, 56.19% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 40m 53s. Estimated total time: 19h 45m 18s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 33s.
[2025-11-13 09:08:14,101][__main__][INFO] - Starting iteration 181.
[2025-11-13 09:08:14,104][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:08:14,104][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:08:22,395][__main__][INFO] - Number of regex retries in iteration 181: 0
[2025-11-13 09:08:22,396][__main__][INFO] - agents played in iteration 181 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:08:22,843][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:22,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:22,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:23,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:23,321][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:08:23,321][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:24,031][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:08:24,327][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:08:24,664][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:08:24,991][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:08:25,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:08:25,644][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:08:25,970][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:08:26,296][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:08:26,623][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:08:26,957][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:08:27,281][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:08:27,609][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:08:27,936][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:08:28,263][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:08:28,588][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:08:28,915][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:08:29,242][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:08:29,575][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:08:29,895][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:08:30,222][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:08:30,549][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:08:30,881][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:08:31,204][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:08:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:08:31,857][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:08:32,187][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:08:32,509][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:08:32,836][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:08:33,163][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:08:33,492][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:08:33,824][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:08:34,154][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:08:34,488][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:08:35,260][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:08:36,015][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:36,016][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:36,019][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:08:37,086][__main__][INFO] - Iteration 182 took 22s (36.07% Gen, 59.28% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 4m 20s. Estimated total time: 19h 9m 9s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 18s, 500 more iterations: 3h 11m 31s.
[2025-11-13 09:08:37,088][__main__][INFO] - Starting iteration 182.
[2025-11-13 09:08:37,091][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:08:37,092][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:08:45,648][__main__][INFO] - Number of regex retries in iteration 182: 0
[2025-11-13 09:08:45,648][__main__][INFO] - agents played in iteration 182 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:08:46,120][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,240][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:08:46,241][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:08:46,241][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:08:47,032][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:08:47,330][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:08:47,658][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:08:47,988][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:08:48,314][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:08:48,641][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:08:48,968][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:08:49,295][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:08:49,621][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:08:49,948][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:08:50,274][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:08:50,600][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:08:50,927][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:08:51,254][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:08:51,583][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:08:51,913][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:08:52,239][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:08:52,566][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:08:52,892][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:08:53,219][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:08:53,555][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:08:53,883][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:08:54,209][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:08:54,535][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:08:54,872][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:08:55,199][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:08:55,525][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:08:55,852][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:08:56,183][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:08:56,510][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:08:56,838][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:08:57,166][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:08:57,492][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:08:58,246][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:08:59,031][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:08:59,033][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:08:59,034][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:00,037][__main__][INFO] - Iteration 183 took 22s (37.29% Gen, 58.34% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 2m 9s. Estimated total time: 19h 7m 20s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 14s, 500 more iterations: 3h 11m 13s.
[2025-11-13 09:09:00,039][__main__][INFO] - Starting iteration 183.
[2025-11-13 09:09:00,042][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:09:00,043][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:08,040][__main__][INFO] - Number of regex retries in iteration 183: 0
[2025-11-13 09:09:08,041][__main__][INFO] - agents played in iteration 183 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:09:08,503][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:08,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:08,584][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:08,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:08,624][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:08,624][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:09,414][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:09:09,713][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:09:10,040][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:09:10,369][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:09:10,696][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:09:11,023][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:09:11,349][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:09:11,676][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:09:12,001][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:09:12,327][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:09:12,654][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:09:12,981][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:09:13,308][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:09:13,641][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:09:13,969][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:09:14,299][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:09:14,625][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:09:14,952][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:09:15,278][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:09:15,604][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:09:15,931][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:09:16,258][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:09:16,584][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:09:16,910][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:09:17,237][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:09:17,563][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:09:17,891][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:09:18,217][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:09:18,546][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:09:18,873][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:09:19,202][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:09:19,529][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:09:19,863][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:09:20,582][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:09:21,326][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:09:21,328][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:09:21,330][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:22,340][__main__][INFO] - Iteration 184 took 22s (35.86% Gen, 59.60% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 29m 23s. Estimated total time: 18h 34m 57s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 9s, 500 more iterations: 3h 5m 49s.
[2025-11-13 09:09:22,342][__main__][INFO] - Starting iteration 184.
[2025-11-13 09:09:22,346][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:09:22,346][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:30,978][__main__][INFO] - Number of regex retries in iteration 184: 0
[2025-11-13 09:09:30,978][__main__][INFO] - agents played in iteration 184 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:09:31,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:31,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:31,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:31,588][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:31,589][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:31,589][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:32,352][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:09:32,650][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:09:32,987][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:09:33,314][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:09:33,642][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:09:33,974][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:09:34,303][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:09:34,633][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:09:34,959][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:09:35,286][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:09:35,614][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:09:35,941][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:09:36,268][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:09:36,599][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:09:36,921][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:09:37,246][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:09:37,574][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:09:37,903][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:09:38,227][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:09:38,552][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:09:38,880][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:09:39,207][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:09:39,533][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:09:39,860][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:09:40,186][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:09:40,512][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:09:40,838][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:09:41,164][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:09:41,491][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:09:41,819][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:09:42,146][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:09:42,473][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:09:42,800][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:09:43,513][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:09:44,255][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:09:44,256][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:09:44,258][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:09:45,263][__main__][INFO] - Iteration 185 took 22s (37.66% Gen, 57.95% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 59m 57s. Estimated total time: 19h 5m 54s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 11s, 500 more iterations: 3h 10m 59s.
[2025-11-13 09:09:45,265][__main__][INFO] - Starting iteration 185.
[2025-11-13 09:09:45,269][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:09:45,270][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:09:53,713][__main__][INFO] - Number of regex retries in iteration 185: 0
[2025-11-13 09:09:53,714][__main__][INFO] - agents played in iteration 185 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:09:54,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:54,212][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:54,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:54,293][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:09:54,293][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:09:54,294][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:09:55,053][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:09:55,351][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:09:55,678][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:09:56,003][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:09:56,331][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:09:56,657][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:09:56,984][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:09:57,311][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:09:57,644][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:09:57,975][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:09:58,302][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:09:58,635][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:09:58,965][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:09:59,292][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:09:59,618][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:09:59,952][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:10:00,286][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:10:00,617][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:10:00,946][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:10:01,275][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:10:01,603][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:10:01,929][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:10:02,255][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:10:02,585][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:10:02,913][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:10:03,242][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:10:03,571][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:10:03,904][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:10:04,233][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:10:04,566][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:10:04,891][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:10:05,224][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:10:05,551][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:10:06,285][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:10:07,052][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:07,053][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:07,055][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:08,046][__main__][INFO] - Iteration 186 took 22s (37.07% Gen, 58.57% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 52m 33s. Estimated total time: 18h 58m 53s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 57s, 500 more iterations: 3h 9m 48s.
[2025-11-13 09:10:08,048][__main__][INFO] - Starting iteration 186.
[2025-11-13 09:10:08,052][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:10:08,053][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:10:16,307][__main__][INFO] - Number of regex retries in iteration 186: 0
[2025-11-13 09:10:16,308][__main__][INFO] - agents played in iteration 186 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:10:16,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:16,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:16,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:16,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:16,886][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:10:16,886][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:10:17,644][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:10:17,943][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:10:18,273][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:10:18,604][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:10:18,934][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:10:19,269][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:10:19,602][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:10:19,931][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:10:20,258][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:10:20,586][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:10:20,915][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:10:21,243][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:10:21,569][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:10:21,897][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:10:22,225][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:10:22,552][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:10:22,880][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:10:23,206][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:10:23,532][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:10:23,859][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:10:24,194][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:10:24,519][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:10:24,846][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:10:25,173][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:10:25,501][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:10:25,828][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:10:26,156][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:10:26,487][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:10:26,811][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:10:27,138][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:10:27,467][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:10:27,797][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:10:28,122][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:10:28,826][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:10:29,605][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:29,606][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:29,608][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:30,641][__main__][INFO] - Iteration 187 took 22s (36.54% Gen, 58.88% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 42m 48s. Estimated total time: 18h 49m 30s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 39s, 500 more iterations: 3h 8m 15s.
[2025-11-13 09:10:30,643][__main__][INFO] - Starting iteration 187.
[2025-11-13 09:10:30,646][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:10:30,647][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:10:38,555][__main__][INFO] - Number of regex retries in iteration 187: 0
[2025-11-13 09:10:38,556][__main__][INFO] - agents played in iteration 187 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:10:39,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:39,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:39,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:39,132][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:10:39,133][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:10:39,133][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:10:39,893][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:10:40,199][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:10:40,528][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:10:40,855][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:10:41,183][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:10:41,510][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:10:41,838][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:10:42,164][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:10:42,491][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:10:42,820][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:10:43,146][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:10:43,473][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:10:43,800][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:10:44,129][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:10:44,456][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:10:44,783][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:10:45,111][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:10:45,437][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:10:45,765][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:10:46,094][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:10:46,422][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:10:46,750][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:10:47,076][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:10:47,403][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:10:47,730][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:10:48,056][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:10:48,383][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:10:48,713][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:10:49,039][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:10:49,365][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:10:49,693][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:10:50,019][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:10:50,346][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:10:51,025][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:10:51,787][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:10:51,788][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:10:51,790][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:10:52,787][__main__][INFO] - Iteration 188 took 22s (35.72% Gen, 59.77% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 20m 2s. Estimated total time: 18h 27m 5s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 54s, 500 more iterations: 3h 4m 30s.
[2025-11-13 09:10:52,790][__main__][INFO] - Starting iteration 188.
[2025-11-13 09:10:52,793][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:10:52,794][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:01,040][__main__][INFO] - Number of regex retries in iteration 188: 0
[2025-11-13 09:11:01,041][__main__][INFO] - agents played in iteration 188 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:11:01,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:01,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:01,578][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:01,617][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:01,618][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:01,618][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:02,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:11:02,782][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:11:03,126][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:11:03,456][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:11:03,790][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:11:04,118][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:11:04,445][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:11:04,771][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:11:05,099][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:11:05,427][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:11:05,756][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:11:06,083][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:11:06,409][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:11:06,736][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:11:07,064][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:11:07,392][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:11:07,718][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:11:08,044][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:11:08,373][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:11:08,699][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:11:09,028][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:11:09,355][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:11:09,682][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:11:10,008][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:11:10,336][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:11:10,663][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:11:10,997][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:11:11,327][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:11:11,652][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:11:11,979][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:11:12,306][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:11:12,631][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:11:12,958][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:11:13,642][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:11:14,408][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:11:14,410][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:11:14,412][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:11:15,523][__main__][INFO] - Iteration 189 took 22s (36.28% Gen, 58.82% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 49m 4s. Estimated total time: 18h 56m 31s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 53s, 500 more iterations: 3h 9m 25s.
[2025-11-13 09:11:15,525][__main__][INFO] - Starting iteration 189.
[2025-11-13 09:11:15,528][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:11:15,528][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:23,534][__main__][INFO] - Number of regex retries in iteration 189: 0
[2025-11-13 09:11:23,534][__main__][INFO] - agents played in iteration 189 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:11:23,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:24,030][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:24,069][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:24,109][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:24,109][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:24,110][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:24,894][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:11:25,193][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:11:25,520][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:11:25,848][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:11:26,180][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:11:26,511][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:11:26,838][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:11:27,166][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:11:27,495][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:11:27,821][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:11:28,148][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:11:28,475][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:11:28,812][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:11:29,144][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:11:29,473][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:11:29,802][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:11:30,127][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:11:30,457][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:11:30,782][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:11:31,112][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:11:31,440][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:11:31,768][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:11:32,094][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:11:32,425][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:11:32,757][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:11:33,084][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:11:33,410][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:11:33,737][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:11:34,064][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:11:34,392][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:11:34,720][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:11:35,046][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:11:35,372][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:11:36,058][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:11:36,811][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:11:36,812][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:11:36,814][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:11:37,994][__main__][INFO] - Iteration 190 took 22s (35.63% Gen, 59.11% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 35m 30s. Estimated total time: 18h 43m 20s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 26s, 500 more iterations: 3h 7m 13s.
[2025-11-13 09:11:37,996][__main__][INFO] - Starting iteration 190.
[2025-11-13 09:11:38,000][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 18 and human policies 1.
[2025-11-13 09:11:38,001][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:11:45,919][__main__][INFO] - Number of regex retries in iteration 190: 0
[2025-11-13 09:11:45,920][__main__][INFO] - agents played in iteration 190 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:11:46,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:46,416][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:46,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:46,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:11:46,496][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:11:46,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:11:47,277][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:11:47,576][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:11:47,902][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:11:48,228][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:11:48,555][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:11:48,882][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:11:49,210][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:11:49,538][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:11:49,879][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:11:50,208][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:11:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:11:50,870][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:11:51,208][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:11:51,535][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:11:51,862][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:11:52,188][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:11:52,518][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:11:52,844][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:11:53,170][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:11:53,497][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:11:53,826][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:11:54,152][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:11:54,478][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:11:54,805][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:11:55,132][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:11:55,458][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:11:55,785][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:11:56,111][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:11:56,437][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:11:56,763][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:11:57,090][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:11:57,421][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:11:57,744][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:11:58,436][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:11:59,193][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:11:59,195][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:11:59,196][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:12:01,270][__main__][INFO] - Iteration 191 took 23s (34.03% Gen, 57.05% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 15m 21s. Estimated total time: 19h 23m 33s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 47s, 500 more iterations: 3h 13m 55s.
[2025-11-13 09:12:01,272][__main__][INFO] - Starting iteration 191.
[2025-11-13 09:12:01,276][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:12:01,276][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:10,072][__main__][INFO] - Number of regex retries in iteration 191: 0
[2025-11-13 09:12:10,072][__main__][INFO] - agents played in iteration 191 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:12:10,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:10,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:10,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:10,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:10,664][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:10,665][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:11,421][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:12:11,718][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:12:12,047][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:12:12,377][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:12:12,707][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:12:13,035][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:12:13,365][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:12:13,691][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:12:14,017][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:12:14,343][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:12:14,673][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:12:14,998][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:12:15,325][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:12:15,655][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:12:15,984][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:12:16,310][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:12:16,637][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:12:16,963][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:12:17,298][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:12:17,627][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:12:17,954][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:12:18,280][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:12:18,619][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:12:18,945][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:12:19,274][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:12:19,600][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:12:19,927][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:12:20,255][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:12:20,582][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:12:20,911][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:12:21,237][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:12:21,566][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:12:21,893][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:12:22,628][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:12:23,381][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:12:23,382][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:12:23,384][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:12:24,390][__main__][INFO] - Iteration 192 took 23s (38.05% Gen, 57.59% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 7m 11s. Estimated total time: 19h 15m 46s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 31s, 500 more iterations: 3h 12m 37s.
[2025-11-13 09:12:24,392][__main__][INFO] - Starting iteration 192.
[2025-11-13 09:12:24,396][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:12:24,397][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:32,894][__main__][INFO] - Number of regex retries in iteration 192: 0
[2025-11-13 09:12:32,894][__main__][INFO] - agents played in iteration 192 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:12:33,342][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:33,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:33,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:33,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:33,462][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:33,462][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:34,231][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:12:34,529][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:12:34,860][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:12:35,189][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:12:35,521][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:12:35,849][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:12:36,180][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:12:36,509][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:12:36,841][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:12:37,165][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:12:37,492][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:12:37,818][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:12:38,144][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:12:38,470][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:12:38,796][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:12:39,124][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:12:39,455][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:12:39,781][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:12:40,106][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:12:40,432][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:12:40,758][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:12:41,085][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:12:41,412][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:12:41,741][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:12:42,075][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:12:42,408][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:12:42,737][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:12:43,066][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:12:43,393][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:12:43,722][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:12:44,052][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:12:44,378][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:12:44,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:12:45,397][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:12:46,134][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:12:46,135][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:12:46,137][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:12:47,131][__main__][INFO] - Iteration 193 took 22s (37.37% Gen, 58.25% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 47m 50s. Estimated total time: 18h 56m 48s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 53s, 500 more iterations: 3h 9m 28s.
[2025-11-13 09:12:47,133][__main__][INFO] - Starting iteration 193.
[2025-11-13 09:12:47,137][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:12:47,138][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:12:54,980][__main__][INFO] - Number of regex retries in iteration 193: 0
[2025-11-13 09:12:54,980][__main__][INFO] - agents played in iteration 193 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:12:55,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:55,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:55,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:55,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:12:55,553][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:12:55,553][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:12:56,292][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:12:56,589][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:12:56,918][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:12:57,245][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:12:57,576][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:12:57,916][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:12:58,246][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:12:58,572][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:12:58,902][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:12:59,238][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:12:59,565][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:12:59,895][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:13:00,229][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:13:00,553][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:13:00,883][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:13:01,213][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:13:01,540][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:13:01,866][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:13:02,198][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:13:02,525][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:13:02,859][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:13:03,185][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:13:03,511][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:13:03,838][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:13:04,164][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:13:04,494][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:13:04,821][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:13:05,154][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:13:05,483][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:13:05,821][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:13:06,148][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:13:06,474][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:13:06,805][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:13:07,500][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:13:08,230][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:08,232][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:08,234][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:09,234][__main__][INFO] - Iteration 194 took 22s (35.49% Gen, 59.98% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 15m 32s. Estimated total time: 18h 24m 53s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 49s, 500 more iterations: 3h 4m 8s.
[2025-11-13 09:13:09,236][__main__][INFO] - Starting iteration 194.
[2025-11-13 09:13:09,239][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:13:09,240][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:13:17,063][__main__][INFO] - Number of regex retries in iteration 194: 0
[2025-11-13 09:13:17,064][__main__][INFO] - agents played in iteration 194 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:13:17,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:17,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:17,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:17,650][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:17,651][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:13:17,651][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:13:18,404][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:13:18,702][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:13:19,029][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:13:19,358][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:13:19,692][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:13:20,019][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:13:20,346][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:13:20,673][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:13:21,002][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:13:21,334][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:13:21,658][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:13:21,986][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:13:22,316][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:13:22,643][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:13:22,970][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:13:23,299][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:13:23,628][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:13:23,958][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:13:24,289][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:13:24,621][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:13:24,951][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:13:25,281][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:13:25,611][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:13:25,946][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:13:26,280][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:13:26,606][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:13:26,939][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:13:27,269][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:13:27,597][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:13:27,925][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:13:28,252][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:13:28,579][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:13:28,911][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:13:29,612][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:13:30,345][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:30,347][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:30,348][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:31,353][__main__][INFO] - Iteration 195 took 22s (35.38% Gen, 60.07% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 16m 1s. Estimated total time: 18h 25m 43s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 51s, 500 more iterations: 3h 4m 17s.
[2025-11-13 09:13:31,355][__main__][INFO] - Starting iteration 195.
[2025-11-13 09:13:31,359][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:13:31,360][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:13:39,206][__main__][INFO] - Number of regex retries in iteration 195: 0
[2025-11-13 09:13:39,206][__main__][INFO] - agents played in iteration 195 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:13:39,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,728][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,807][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:13:39,808][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:13:39,808][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:13:40,561][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:13:40,857][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:13:41,184][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:13:41,511][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:13:41,838][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:13:42,167][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:13:42,498][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:13:42,825][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:13:43,152][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:13:43,479][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:13:43,810][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:13:44,136][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:13:44,465][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:13:44,792][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:13:45,122][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:13:45,452][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:13:45,781][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:13:46,108][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:13:46,438][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:13:46,772][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:13:47,100][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:13:47,433][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:13:47,760][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:13:48,087][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:13:48,413][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:13:48,739][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:13:49,069][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:13:49,397][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:13:49,725][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:13:50,052][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:13:50,379][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:13:50,705][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:13:51,033][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:13:51,721][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:13:52,459][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:13:52,463][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:13:52,465][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:13:53,531][__main__][INFO] - Iteration 196 took 22s (35.39% Gen, 59.80% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 18m 32s. Estimated total time: 18h 28m 37s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 57s, 500 more iterations: 3h 4m 46s.
[2025-11-13 09:13:53,533][__main__][INFO] - Starting iteration 196.
[2025-11-13 09:13:53,536][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:13:53,536][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:01,028][__main__][INFO] - Number of regex retries in iteration 196: 0
[2025-11-13 09:14:01,029][__main__][INFO] - agents played in iteration 196 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:14:01,484][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:01,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:01,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:01,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:01,604][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:01,605][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:02,365][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:14:02,800][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:14:03,128][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:14:03,459][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:14:03,785][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:14:04,115][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:14:04,445][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:14:04,773][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:14:05,100][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:14:05,435][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:14:05,762][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:14:06,089][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:14:06,420][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:14:06,749][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:14:07,082][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:14:07,410][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:14:07,738][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:14:08,063][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:14:08,394][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:14:08,736][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:14:09,063][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:14:09,392][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:14:09,720][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:14:10,047][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:14:10,373][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:14:10,707][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:14:11,033][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:14:11,366][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:14:11,693][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:14:12,021][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:14:12,354][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:14:12,676][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:14:13,002][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:14:13,688][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:14:14,443][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:14:14,445][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:14:14,446][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:14:15,456][__main__][INFO] - Iteration 197 took 21s (34.18% Gen, 61.21% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 5m 36s. Estimated total time: 18h 16m 2s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 32s, 500 more iterations: 3h 2m 40s.
[2025-11-13 09:14:15,458][__main__][INFO] - Starting iteration 197.
[2025-11-13 09:14:15,462][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:14:15,462][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:23,094][__main__][INFO] - Number of regex retries in iteration 197: 0
[2025-11-13 09:14:23,095][__main__][INFO] - agents played in iteration 197 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:14:23,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:23,589][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:23,629][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:23,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:23,670][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:23,670][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:24,456][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:14:24,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:14:25,086][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:14:25,415][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:14:25,741][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:14:26,068][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:14:26,396][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:14:26,723][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:14:27,057][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:14:27,386][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:14:27,715][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:14:28,048][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:14:28,376][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:14:28,703][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:14:29,032][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:14:29,365][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:14:29,695][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:14:30,023][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:14:30,350][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:14:30,676][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:14:31,003][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:14:31,336][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:14:31,665][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:14:31,992][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:14:32,318][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:14:32,659][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:14:32,985][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:14:33,312][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:14:33,640][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:14:33,972][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:14:34,298][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:14:34,624][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:14:34,955][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:14:35,636][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:14:36,378][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:14:36,379][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:14:36,381][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:14:37,466][__main__][INFO] - Iteration 198 took 22s (34.68% Gen, 60.38% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 9m 27s. Estimated total time: 18h 20m 16s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 40s, 500 more iterations: 3h 3m 22s.
[2025-11-13 09:14:37,469][__main__][INFO] - Starting iteration 198.
[2025-11-13 09:14:37,472][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:14:37,473][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:14:45,077][__main__][INFO] - Number of regex retries in iteration 198: 0
[2025-11-13 09:14:45,078][__main__][INFO] - agents played in iteration 198 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:14:45,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:45,585][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:45,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:45,664][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:14:45,665][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:14:45,666][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:14:46,429][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:14:46,728][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:14:47,055][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:14:47,383][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:14:47,710][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:14:48,035][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:14:48,367][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:14:48,694][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:14:49,024][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:14:49,362][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:14:49,686][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:14:50,014][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:14:50,343][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:14:50,675][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:14:50,999][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:14:51,329][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:14:51,656][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:14:51,982][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:14:52,312][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:14:52,644][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:14:52,973][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:14:53,299][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:14:53,625][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:14:53,955][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:14:54,287][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:14:54,616][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:14:54,953][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:14:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:14:55,622][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:14:55,960][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:14:56,285][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:14:56,614][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:14:56,944][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:14:57,632][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:14:58,358][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:14:58,359][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:14:58,361][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:14:59,418][__main__][INFO] - Iteration 199 took 21s (34.65% Gen, 60.53% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 6m 10s. Estimated total time: 18h 17m 21s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 34s, 500 more iterations: 3h 2m 53s.
[2025-11-13 09:14:59,420][__main__][INFO] - Starting iteration 199.
[2025-11-13 09:14:59,423][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:14:59,424][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:07,013][__main__][INFO] - Number of regex retries in iteration 199: 0
[2025-11-13 09:15:07,014][__main__][INFO] - agents played in iteration 199 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:15:07,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:07,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:07,557][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:07,597][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:07,598][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:07,598][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:08,394][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:15:08,692][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:15:09,019][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:15:09,360][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:15:09,690][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:15:10,018][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:15:10,345][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:15:10,681][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:15:11,017][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:15:11,351][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:15:11,682][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:15:12,007][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:15:12,334][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:15:12,667][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:15:12,998][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:15:13,339][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:15:13,666][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:15:13,996][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:15:14,324][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:15:14,651][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:15:14,979][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:15:15,306][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:15:15,632][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:15:15,958][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:15:16,288][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:15:16,615][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:15:16,942][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:15:17,274][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:15:17,614][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:15:17,944][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:15:18,275][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:15:18,599][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:15:18,925][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:15:19,626][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:15:20,367][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:15:20,369][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:15:20,370][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:15:21,365][__main__][INFO] - Iteration 200 took 21s (34.59% Gen, 60.87% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 5m 35s. Estimated total time: 18h 17m 7s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 34s, 500 more iterations: 3h 2m 51s.
[2025-11-13 09:15:21,367][__main__][INFO] - Starting iteration 200.
[2025-11-13 09:15:21,371][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 19 and human policies 1.
[2025-11-13 09:15:21,372][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:29,067][__main__][INFO] - Number of regex retries in iteration 200: 0
[2025-11-13 09:15:29,067][__main__][INFO] - agents played in iteration 200 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:15:29,522][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:29,562][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:29,602][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:29,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:29,643][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:29,644][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:30,415][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:15:30,713][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:15:31,041][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:15:31,370][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:15:31,697][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:15:32,025][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:15:32,357][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:15:32,687][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:15:33,016][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:15:33,346][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:15:33,674][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:15:34,002][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:15:34,329][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:15:34,659][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:15:34,987][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:15:35,314][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:15:35,644][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:15:35,972][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:15:36,306][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:15:36,635][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:15:36,968][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:15:37,295][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:15:37,622][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:15:37,952][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:15:38,279][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:15:38,604][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:15:38,933][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:15:39,259][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:15:39,586][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:15:39,914][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:15:40,241][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:15:40,571][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:15:40,896][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:15:41,577][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:15:42,468][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:15:42,469][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:15:42,481][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:15:44,993][__main__][INFO] - Iteration 201 took 23s (32.57% Gen, 56.79% Train). Generation: 7s, Training: 13s. Estimated remaining time: 18h 29m 12s. Estimated total time: 19h 41m 8s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 22s, 500 more iterations: 3h 16m 51s.
[2025-11-13 09:15:44,995][__main__][INFO] - Starting iteration 201.
[2025-11-13 09:15:44,998][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:15:44,999][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:15:52,644][__main__][INFO] - Number of regex retries in iteration 201: 0
[2025-11-13 09:15:52,645][__main__][INFO] - agents played in iteration 201 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:15:53,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:53,150][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:53,190][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:53,595][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:15:53,596][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:15:53,597][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:15:54,364][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:15:54,662][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:15:54,989][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:15:55,322][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:15:55,648][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:15:55,979][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:15:56,313][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:15:56,643][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:15:56,973][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:15:57,303][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:15:57,633][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:15:57,967][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:15:58,295][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:15:58,622][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:15:58,949][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:15:59,277][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:15:59,604][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:15:59,934][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:16:00,260][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:16:00,588][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:16:00,915][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:16:01,242][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:16:01,570][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:16:01,896][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:16:02,226][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:16:02,556][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:16:02,884][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:16:03,211][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:16:03,537][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:16:03,864][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:16:04,198][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:16:04,525][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:16:04,853][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:16:05,553][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:16:06,295][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:06,297][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:06,299][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:07,381][__main__][INFO] - Iteration 202 took 22s (34.16% Gen, 61.00% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 26m 51s. Estimated total time: 18h 39m 9s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 18s, 500 more iterations: 3h 6m 31s.
[2025-11-13 09:16:07,383][__main__][INFO] - Starting iteration 202.
[2025-11-13 09:16:07,386][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:16:07,386][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:16:14,967][__main__][INFO] - Number of regex retries in iteration 202: 0
[2025-11-13 09:16:14,967][__main__][INFO] - agents played in iteration 202 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:16:15,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:15,464][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:15,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:15,544][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:15,544][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:16:15,545][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:16:16,320][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:16:16,617][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:16:16,945][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:16:17,278][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:16:17,605][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:16:17,933][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:16:18,260][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:16:18,589][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:16:18,917][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:16:19,247][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:16:19,575][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:16:19,904][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:16:20,231][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:16:20,558][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:16:20,885][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:16:21,213][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:16:21,540][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:16:21,867][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:16:22,194][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:16:22,522][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:16:22,848][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:16:23,175][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:16:23,504][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:16:23,831][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:16:24,160][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:16:24,485][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:16:24,812][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:16:25,139][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:16:25,466][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:16:25,799][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:16:26,125][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:16:26,451][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:16:26,777][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:16:27,461][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:16:28,191][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:28,193][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:28,194][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:29,236][__main__][INFO] - Iteration 203 took 21s (34.70% Gen, 60.53% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 59m 53s. Estimated total time: 18h 12m 33s. Time estimates for 10 more iterations: 3m 38s, 100 more iterations: 36m 25s, 500 more iterations: 3h 2m 5s.
[2025-11-13 09:16:29,238][__main__][INFO] - Starting iteration 203.
[2025-11-13 09:16:29,242][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:16:29,242][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:16:36,580][__main__][INFO] - Number of regex retries in iteration 203: 0
[2025-11-13 09:16:36,581][__main__][INFO] - agents played in iteration 203 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:16:37,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:37,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:37,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:37,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:37,175][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:16:37,175][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:16:37,940][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:16:38,239][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:16:38,566][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:16:38,893][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:16:39,221][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:16:39,549][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:16:39,877][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:16:40,205][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:16:40,534][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:16:40,862][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:16:41,189][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:16:41,517][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:16:41,846][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:16:42,174][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:16:42,502][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:16:42,829][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:16:43,156][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:16:43,487][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:16:43,814][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:16:44,141][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:16:44,468][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:16:44,804][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:16:45,131][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:16:45,459][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:16:45,787][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:16:46,115][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:16:46,442][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:16:46,768][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:16:47,095][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:16:47,422][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:16:47,749][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:16:48,075][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:16:48,401][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:16:49,096][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:16:49,878][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:16:49,880][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:16:49,884][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:16:51,165][__main__][INFO] - Iteration 204 took 21s (33.47% Gen, 60.68% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 3m 12s. Estimated total time: 18h 16m 14s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 32s, 500 more iterations: 3h 2m 42s.
[2025-11-13 09:16:51,168][__main__][INFO] - Starting iteration 204.
[2025-11-13 09:16:51,171][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:16:51,171][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:16:58,163][__main__][INFO] - Number of regex retries in iteration 204: 0
[2025-11-13 09:16:58,164][__main__][INFO] - agents played in iteration 204 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:16:58,625][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:58,665][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:58,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:58,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:16:58,747][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:16:58,747][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:16:59,540][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:16:59,838][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:17:00,166][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:17:00,492][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:17:00,819][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:17:01,146][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:17:01,474][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:17:01,802][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:17:02,130][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:17:02,457][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:17:02,785][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:17:03,114][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:17:03,441][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:17:03,770][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:17:04,098][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:17:04,424][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:17:04,751][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:17:05,078][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:17:05,405][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:17:05,732][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:17:06,061][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:17:06,389][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:17:06,716][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:17:07,043][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:17:07,369][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:17:07,697][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:17:08,026][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:17:08,357][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:17:08,685][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:17:09,015][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:17:09,341][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:17:09,668][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:17:09,994][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:17:10,686][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:17:11,440][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:17:11,442][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:17:11,443][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:17:12,453][__main__][INFO] - Iteration 205 took 21s (32.85% Gen, 62.39% Train). Generation: 6s, Training: 13s. Estimated remaining time: 16h 30m 46s. Estimated total time: 17h 44m 10s. Time estimates for 10 more iterations: 3m 32s, 100 more iterations: 35m 28s, 500 more iterations: 2h 57m 21s.
[2025-11-13 09:17:12,456][__main__][INFO] - Starting iteration 205.
[2025-11-13 09:17:12,460][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:17:12,460][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:17:20,211][__main__][INFO] - Number of regex retries in iteration 205: 0
[2025-11-13 09:17:20,211][__main__][INFO] - agents played in iteration 205 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:17:20,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:20,719][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:20,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:20,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:20,800][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:17:20,800][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:17:21,580][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:17:21,880][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:17:22,208][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:17:22,534][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:17:22,862][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:17:23,189][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:17:23,516][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:17:23,843][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:17:24,171][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:17:24,498][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:17:24,825][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:17:25,151][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:17:25,478][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:17:25,807][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:17:26,134][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:17:26,461][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:17:26,787][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:17:27,114][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:17:27,440][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:17:27,766][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:17:28,094][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:17:28,425][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:17:28,751][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:17:29,077][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:17:29,405][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:17:29,734][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:17:30,060][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:17:30,388][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:17:30,715][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:17:31,041][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:17:31,369][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:17:31,696][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:17:32,033][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:17:32,720][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:17:33,467][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:17:33,469][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:17:33,470][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:17:34,664][__main__][INFO] - Iteration 206 took 22s (34.91% Gen, 59.71% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 16m 31s. Estimated total time: 18h 30m 17s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 0s, 500 more iterations: 3h 5m 2s.
[2025-11-13 09:17:34,666][__main__][INFO] - Starting iteration 206.
[2025-11-13 09:17:34,670][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:17:34,670][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:17:42,429][__main__][INFO] - Number of regex retries in iteration 206: 0
[2025-11-13 09:17:42,430][__main__][INFO] - agents played in iteration 206 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:17:42,889][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:42,931][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:42,971][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:43,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:17:43,012][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:17:43,012][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:17:43,807][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:17:44,115][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:17:44,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:17:44,773][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:17:45,098][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:17:45,433][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:17:45,761][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:17:46,088][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:17:46,415][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:17:46,742][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:17:47,069][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:17:47,397][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:17:47,729][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:17:48,050][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:17:48,377][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:17:48,704][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:17:49,037][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:17:49,365][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:17:49,695][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:17:50,024][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:17:50,352][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:17:50,679][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:17:51,006][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:17:51,334][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:17:51,663][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:17:51,992][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:17:52,318][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:17:52,644][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:17:52,973][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:17:53,301][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:17:53,627][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:17:53,956][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:17:54,282][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:17:54,958][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:17:55,687][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:17:55,689][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:17:55,690][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:17:56,697][__main__][INFO] - Iteration 207 took 22s (35.22% Gen, 60.20% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 7m 17s. Estimated total time: 18h 21m 25s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 42s, 500 more iterations: 3h 3m 34s.
[2025-11-13 09:17:56,699][__main__][INFO] - Starting iteration 207.
[2025-11-13 09:17:56,702][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:17:56,703][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:04,400][__main__][INFO] - Number of regex retries in iteration 207: 0
[2025-11-13 09:18:04,401][__main__][INFO] - agents played in iteration 207 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:18:04,873][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:04,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:04,953][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:04,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:04,994][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:18:04,995][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:18:05,767][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:18:06,065][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:18:06,395][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:18:06,724][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:18:07,058][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:18:07,385][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:18:07,725][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:18:08,052][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:18:08,380][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:18:08,709][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:18:09,038][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:18:09,366][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:18:09,694][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:18:10,020][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:18:10,348][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:18:10,675][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:18:11,001][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:18:11,328][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:18:11,655][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:18:11,988][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:18:12,314][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:18:12,640][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:18:12,966][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:18:13,297][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:18:13,623][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:18:13,950][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:18:14,277][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:18:14,603][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:18:14,933][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:18:15,260][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:18:15,588][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:18:15,916][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:18:16,242][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:18:16,965][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:18:17,710][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:18:17,712][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:18:17,713][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:18:18,758][__main__][INFO] - Iteration 208 took 22s (34.90% Gen, 60.36% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 8m 18s. Estimated total time: 18h 22m 48s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 45s, 500 more iterations: 3h 3m 48s.
[2025-11-13 09:18:18,760][__main__][INFO] - Starting iteration 208.
[2025-11-13 09:18:18,763][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:18:18,764][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:26,052][__main__][INFO] - Number of regex retries in iteration 208: 0
[2025-11-13 09:18:26,052][__main__][INFO] - agents played in iteration 208 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:18:26,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:26,542][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:26,582][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:26,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:26,623][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:18:26,623][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:18:27,384][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:18:27,682][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:18:28,011][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:18:28,346][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:18:28,673][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:18:29,001][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:18:29,331][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:18:29,660][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:18:29,991][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:18:30,319][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:18:30,647][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:18:30,975][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:18:31,312][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:18:31,640][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:18:31,968][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:18:32,294][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:18:32,631][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:18:32,958][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:18:33,285][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:18:33,614][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:18:33,943][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:18:34,270][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:18:34,600][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:18:34,930][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:18:35,253][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:18:35,580][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:18:35,911][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:18:36,244][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:18:36,566][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:18:36,896][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:18:37,223][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:18:37,551][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:18:37,881][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:18:38,620][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:18:39,366][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:18:39,368][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:18:39,370][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:18:40,435][__main__][INFO] - Iteration 209 took 21s (33.63% Gen, 61.45% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 48m 47s. Estimated total time: 18h 3m 39s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 7s, 500 more iterations: 3h 0m 36s.
[2025-11-13 09:18:40,437][__main__][INFO] - Starting iteration 209.
[2025-11-13 09:18:40,441][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:18:40,441][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:18:48,150][__main__][INFO] - Number of regex retries in iteration 209: 0
[2025-11-13 09:18:48,151][__main__][INFO] - agents played in iteration 209 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:18:48,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:48,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:48,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:48,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:18:48,719][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:18:48,720][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:18:49,456][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:18:49,752][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:18:50,079][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:18:50,406][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:18:50,734][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:18:51,062][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:18:51,392][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:18:51,722][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:18:52,054][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:18:52,385][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:18:52,713][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:18:53,041][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:18:53,368][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:18:53,695][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:18:54,022][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:18:54,350][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:18:54,677][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:18:55,005][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:18:55,331][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:18:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:18:55,991][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:18:56,318][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:18:56,646][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:18:56,972][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:18:57,301][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:18:57,632][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:18:57,958][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:18:58,284][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:18:58,621][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:18:58,950][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:18:59,275][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:18:59,602][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:18:59,940][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:19:00,669][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:19:01,432][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:01,433][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:01,435][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:19:02,441][__main__][INFO] - Iteration 210 took 22s (35.04% Gen, 60.38% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 4m 51s. Estimated total time: 18h 20m 4s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 40s, 500 more iterations: 3h 3m 20s.
[2025-11-13 09:19:02,443][__main__][INFO] - Starting iteration 210.
[2025-11-13 09:19:02,447][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 20 and human policies 1.
[2025-11-13 09:19:02,448][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:19:09,765][__main__][INFO] - Number of regex retries in iteration 210: 0
[2025-11-13 09:19:09,765][__main__][INFO] - agents played in iteration 210 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:19:10,227][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:10,348][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:10,348][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:11,097][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:19:11,394][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:19:11,734][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:19:12,060][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:19:12,386][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:19:12,715][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:19:13,046][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:19:13,373][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:19:13,707][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:19:14,040][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:19:14,371][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:19:14,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:19:15,039][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:19:15,366][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:19:15,693][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:19:16,021][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:19:16,348][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:19:16,676][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:19:17,003][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:19:17,330][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:19:17,657][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:19:17,985][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:19:18,312][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:19:18,640][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:19:18,967][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:19:19,294][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:19:19,634][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:19:19,962][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:19:20,292][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:19:20,618][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:19:20,947][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:19:21,274][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:19:21,601][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:19:22,317][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:19:23,062][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:23,063][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:23,065][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:19:25,168][__main__][INFO] - Iteration 211 took 22s (32.20% Gen, 58.53% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 40m 29s. Estimated total time: 18h 56m 6s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 52s, 500 more iterations: 3h 9m 21s.
[2025-11-13 09:19:25,170][__main__][INFO] - Starting iteration 211.
[2025-11-13 09:19:25,174][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:19:25,174][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:19:33,306][__main__][INFO] - Number of regex retries in iteration 211: 0
[2025-11-13 09:19:33,307][__main__][INFO] - agents played in iteration 211 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:19:33,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:33,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:33,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:33,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:33,915][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:33,916][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:34,657][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:19:34,954][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:19:35,283][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:19:35,614][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:19:35,943][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:19:36,272][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:19:36,609][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:19:36,939][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:19:37,269][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:19:37,598][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:19:37,941][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:19:38,269][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:19:38,598][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:19:38,927][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:19:39,258][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:19:39,585][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:19:39,913][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:19:40,240][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:19:40,567][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:19:40,893][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:19:41,221][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:19:41,548][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:19:41,875][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:19:42,202][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:19:42,531][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:19:42,864][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:19:43,193][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:19:43,527][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:19:43,854][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:19:44,183][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:19:44,514][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:19:44,843][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:19:45,172][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:19:46,034][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:19:46,848][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:19:46,851][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:19:46,853][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:19:47,950][__main__][INFO] - Iteration 212 took 22s (35.70% Gen, 59.47% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 42m 53s. Estimated total time: 18h 58m 52s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 57s, 500 more iterations: 3h 9m 48s.
[2025-11-13 09:19:47,952][__main__][INFO] - Starting iteration 212.
[2025-11-13 09:19:47,956][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:19:47,957][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:19:55,386][__main__][INFO] - Number of regex retries in iteration 212: 0
[2025-11-13 09:19:55,387][__main__][INFO] - agents played in iteration 212 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:19:55,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:55,881][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:55,922][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:55,962][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:19:55,962][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:19:55,963][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:19:56,710][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:19:57,008][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:19:57,335][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:19:57,664][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:19:57,991][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:19:58,318][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:19:58,646][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:19:58,973][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:19:59,301][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:19:59,629][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:19:59,957][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:20:00,285][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:20:00,613][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:20:00,943][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:20:01,271][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:20:01,598][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:20:01,925][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:20:02,252][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:20:02,580][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:20:02,906][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:20:03,235][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:20:03,561][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:20:03,888][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:20:04,216][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:20:04,543][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:20:04,870][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:20:05,197][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:20:05,531][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:20:05,857][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:20:06,184][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:20:06,513][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:20:06,842][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:20:07,176][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:20:07,912][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:20:08,673][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:20:08,675][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:20:08,676][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:09,689][__main__][INFO] - Iteration 213 took 21s (34.19% Gen, 61.14% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 50m 21s. Estimated total time: 18h 6m 42s. Time estimates for 10 more iterations: 3m 37s, 100 more iterations: 36m 13s, 500 more iterations: 3h 1m 7s.
[2025-11-13 09:20:09,691][__main__][INFO] - Starting iteration 213.
[2025-11-13 09:20:09,695][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:20:09,696][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:20:17,026][__main__][INFO] - Number of regex retries in iteration 213: 0
[2025-11-13 09:20:17,027][__main__][INFO] - agents played in iteration 213 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:20:17,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:17,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:17,568][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:17,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:17,608][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:20:17,609][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:20:18,358][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:20:18,655][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:20:18,982][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:20:19,311][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:20:19,640][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:20:19,970][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:20:20,296][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:20:20,623][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:20:20,951][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:20:21,279][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:20:21,607][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:20:21,934][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:20:22,262][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:20:22,593][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:20:22,926][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:20:23,253][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:20:23,583][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:20:23,911][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:20:24,240][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:20:24,566][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:20:24,893][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:20:25,224][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:20:25,551][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:20:25,881][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:20:26,208][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:20:26,538][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:20:26,864][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:20:27,192][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:20:27,520][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:20:27,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:20:28,178][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:20:28,505][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:20:28,832][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:20:29,567][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:20:30,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:20:30,318][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:20:30,320][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:31,321][__main__][INFO] - Iteration 214 took 21s (33.90% Gen, 61.46% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 44m 38s. Estimated total time: 18h 1m 20s. Time estimates for 10 more iterations: 3m 36s, 100 more iterations: 36m 2s, 500 more iterations: 3h 0m 13s.
[2025-11-13 09:20:31,323][__main__][INFO] - Starting iteration 214.
[2025-11-13 09:20:31,327][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:20:31,328][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:20:39,044][__main__][INFO] - Number of regex retries in iteration 214: 0
[2025-11-13 09:20:39,044][__main__][INFO] - agents played in iteration 214 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:20:39,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:39,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:39,596][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:39,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:20:39,636][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:20:39,637][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:20:40,384][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:20:40,682][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:20:41,008][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:20:41,342][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:20:41,669][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:20:41,996][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:20:42,323][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:20:42,652][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:20:42,979][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:20:43,306][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:20:43,638][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:20:43,968][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:20:44,299][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:20:44,628][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:20:44,959][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:20:45,285][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:20:45,614][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:20:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:20:46,272][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:20:46,599][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:20:46,927][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:20:47,254][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:20:47,579][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:20:47,908][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:20:48,235][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:20:48,563][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:20:48,894][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:20:49,221][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:20:49,549][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:20:49,877][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:20:50,204][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:20:50,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:20:50,860][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:20:51,603][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:20:52,344][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:20:52,345][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:20:52,347][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:20:53,353][__main__][INFO] - Iteration 215 took 22s (35.03% Gen, 60.39% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 4m 16s. Estimated total time: 18h 21m 21s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 42s, 500 more iterations: 3h 3m 33s.
[2025-11-13 09:20:53,355][__main__][INFO] - Starting iteration 215.
[2025-11-13 09:20:53,359][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:20:53,360][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:21:01,306][__main__][INFO] - Number of regex retries in iteration 215: 0
[2025-11-13 09:21:01,307][__main__][INFO] - agents played in iteration 215 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:21:01,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:01,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:01,857][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:01,897][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:01,897][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:21:01,898][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:21:02,643][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:21:02,940][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:21:03,271][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:21:03,598][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:21:03,925][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:21:04,255][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:21:04,589][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:21:04,918][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:21:05,251][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:21:05,585][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:21:05,912][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:21:06,238][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:21:06,567][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:21:06,896][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:21:07,226][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:21:07,557][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:21:07,889][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:21:08,219][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:21:08,546][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:21:08,875][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:21:09,204][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:21:09,533][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:21:09,862][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:21:10,189][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:21:10,516][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:21:10,844][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:21:11,171][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:21:11,499][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:21:11,826][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:21:12,153][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:21:12,480][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:21:12,808][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:21:13,136][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:21:13,885][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:21:14,633][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:14,635][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:14,637][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:15,681][__main__][INFO] - Iteration 216 took 22s (35.60% Gen, 59.72% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 18m 41s. Estimated total time: 18h 36m 8s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 12s, 500 more iterations: 3h 6m 1s.
[2025-11-13 09:21:15,683][__main__][INFO] - Starting iteration 216.
[2025-11-13 09:21:15,686][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:21:15,687][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:21:23,502][__main__][INFO] - Number of regex retries in iteration 216: 0
[2025-11-13 09:21:23,503][__main__][INFO] - agents played in iteration 216 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:21:23,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:23,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:24,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:24,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:24,074][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:21:24,075][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:21:24,828][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:21:25,124][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:21:25,453][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:21:25,781][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:21:26,108][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:21:26,435][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:21:26,769][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:21:27,100][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:21:27,431][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:21:27,757][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:21:28,084][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:21:28,411][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:21:28,739][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:21:29,068][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:21:29,396][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:21:29,726][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:21:30,053][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:21:30,381][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:21:30,712][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:21:31,046][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:21:31,378][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:21:31,705][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:21:32,033][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:21:32,366][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:21:32,688][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:21:33,015][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:21:33,343][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:21:33,676][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:21:33,998][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:21:34,326][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:21:34,654][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:21:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:21:35,311][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:21:36,059][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:21:36,823][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:36,825][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:36,826][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:37,844][__main__][INFO] - Iteration 217 took 22s (35.27% Gen, 60.13% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 10m 7s. Estimated total time: 18h 27m 56s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 55s, 500 more iterations: 3h 4m 39s.
[2025-11-13 09:21:37,846][__main__][INFO] - Starting iteration 217.
[2025-11-13 09:21:37,850][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:21:37,851][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:21:45,600][__main__][INFO] - Number of regex retries in iteration 217: 0
[2025-11-13 09:21:45,600][__main__][INFO] - agents played in iteration 217 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:21:46,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:46,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:46,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:46,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:21:46,182][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:21:46,183][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:21:46,919][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:21:47,216][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:21:47,543][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:21:47,871][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:21:48,201][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:21:48,528][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:21:48,862][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:21:49,188][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:21:49,516][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:21:49,844][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:21:50,170][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:21:50,499][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:21:50,827][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:21:51,162][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:21:51,491][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:21:51,817][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:21:52,146][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:21:52,480][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:21:52,812][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:21:53,139][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:21:53,466][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:21:53,794][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:21:54,122][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:21:54,450][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:21:54,777][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:21:55,105][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:21:55,432][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:21:55,760][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:21:56,089][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:21:56,414][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:21:56,742][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:21:57,069][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:21:57,397][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:21:58,152][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:21:58,896][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:21:58,897][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:21:58,899][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:21:59,892][__main__][INFO] - Iteration 218 took 22s (35.16% Gen, 60.33% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 3m 56s. Estimated total time: 18h 22m 7s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 44s, 500 more iterations: 3h 3m 41s.
[2025-11-13 09:21:59,894][__main__][INFO] - Starting iteration 218.
[2025-11-13 09:21:59,897][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:21:59,898][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:07,883][__main__][INFO] - Number of regex retries in iteration 218: 0
[2025-11-13 09:22:07,883][__main__][INFO] - agents played in iteration 218 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:22:08,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:08,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:08,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:08,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:08,449][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:08,450][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:09,192][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:22:09,490][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:22:09,817][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:22:10,145][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:22:10,475][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:22:10,804][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:22:11,132][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:22:11,458][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:22:11,788][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:22:12,115][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:22:12,443][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:22:12,770][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:22:13,098][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:22:13,437][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:22:13,769][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:22:14,095][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:22:14,426][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:22:14,767][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:22:15,095][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:22:15,428][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:22:15,762][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:22:16,092][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:22:16,425][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:22:16,753][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:22:17,080][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:22:17,409][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:22:17,737][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:22:18,064][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:22:18,394][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:22:18,722][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:22:19,050][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:22:19,378][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:22:19,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:22:20,461][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:22:21,177][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:22:21,178][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:22:21,180][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:22:22,190][__main__][INFO] - Iteration 219 took 22s (35.82% Gen, 59.64% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 16m 9s. Estimated total time: 18h 34m 43s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 9s, 500 more iterations: 3h 5m 47s.
[2025-11-13 09:22:22,192][__main__][INFO] - Starting iteration 219.
[2025-11-13 09:22:22,195][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:22:22,196][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:29,982][__main__][INFO] - Number of regex retries in iteration 219: 0
[2025-11-13 09:22:29,983][__main__][INFO] - agents played in iteration 219 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:22:30,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:30,479][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:30,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:30,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:30,559][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:30,559][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:31,300][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:22:31,599][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:22:31,924][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:22:32,250][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:22:32,578][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:22:32,906][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:22:33,234][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:22:33,562][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:22:33,889][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:22:34,216][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:22:34,543][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:22:34,871][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:22:35,200][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:22:35,531][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:22:35,859][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:22:36,185][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:22:36,513][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:22:36,839][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:22:37,173][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:22:37,499][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:22:37,827][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:22:38,156][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:22:38,482][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:22:38,812][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:22:39,138][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:22:39,472][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:22:39,798][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:22:40,125][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:22:40,452][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:22:40,785][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:22:41,107][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:22:41,434][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:22:41,762][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:22:42,508][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:22:43,209][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:22:43,210][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:22:43,212][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:22:44,174][__main__][INFO] - Iteration 220 took 21s (35.43% Gen, 60.19% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 0m 3s. Estimated total time: 18h 18m 58s. Time estimates for 10 more iterations: 3m 39s, 100 more iterations: 36m 37s, 500 more iterations: 3h 3m 9s.
[2025-11-13 09:22:44,176][__main__][INFO] - Starting iteration 220.
[2025-11-13 09:22:44,179][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 21 and human policies 1.
[2025-11-13 09:22:44,180][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:22:52,050][__main__][INFO] - Number of regex retries in iteration 220: 0
[2025-11-13 09:22:52,050][__main__][INFO] - agents played in iteration 220 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:22:52,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:52,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:52,599][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:52,639][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:22:52,640][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:22:52,640][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:22:53,375][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:22:53,671][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:22:53,998][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:22:54,324][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:22:54,652][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:22:54,980][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:22:55,310][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:22:55,639][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:22:55,966][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:22:56,293][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:22:56,622][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:22:56,956][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:22:57,282][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:22:57,610][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:22:57,938][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:22:58,271][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:22:58,597][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:22:58,924][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:22:59,256][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:22:59,583][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:22:59,913][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:23:00,240][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:23:00,570][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:23:00,900][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:23:01,228][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:23:01,556][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:23:01,887][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:23:02,215][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:23:02,543][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:23:02,870][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:23:03,204][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:23:03,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:23:03,861][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:23:04,611][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:23:05,324][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:05,325][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:05,327][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:07,225][__main__][INFO] - Iteration 221 took 23s (34.15% Gen, 57.61% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 53m 0s. Estimated total time: 19h 12m 19s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 24s, 500 more iterations: 3h 12m 3s.
[2025-11-13 09:23:07,227][__main__][INFO] - Starting iteration 221.
[2025-11-13 09:23:07,230][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:23:07,231][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:23:15,747][__main__][INFO] - Number of regex retries in iteration 221: 0
[2025-11-13 09:23:15,748][__main__][INFO] - agents played in iteration 221 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:23:16,201][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:16,240][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:16,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:16,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:16,319][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:23:16,320][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:23:17,039][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:23:17,337][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:23:17,677][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:23:18,006][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:23:18,332][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:23:18,660][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:23:18,990][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:23:19,317][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:23:19,644][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:23:19,979][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:23:20,303][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:23:20,629][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:23:20,957][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:23:21,288][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:23:21,619][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:23:21,945][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:23:22,273][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:23:22,603][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:23:22,931][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:23:23,265][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:23:23,595][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:23:23,924][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:23:24,260][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:23:24,588][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:23:24,915][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:23:25,242][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:23:25,582][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:23:25,909][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:23:26,236][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:23:26,563][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:23:26,892][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:23:27,219][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:23:27,547][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:23:28,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:23:29,013][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:29,015][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:29,016][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:30,037][__main__][INFO] - Iteration 222 took 22s (37.34% Gen, 58.18% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 40m 41s. Estimated total time: 19h 0m 22s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 0s, 500 more iterations: 3h 10m 3s.
[2025-11-13 09:23:30,039][__main__][INFO] - Starting iteration 222.
[2025-11-13 09:23:30,043][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:23:30,043][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:23:38,134][__main__][INFO] - Number of regex retries in iteration 222: 0
[2025-11-13 09:23:38,134][__main__][INFO] - agents played in iteration 222 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:23:38,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:38,647][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:38,686][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:38,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:23:38,726][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:23:38,727][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:23:39,450][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:23:39,747][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:23:40,084][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:23:40,411][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:23:40,738][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:23:41,066][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:23:41,399][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:23:41,721][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:23:42,049][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:23:42,376][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:23:42,706][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:23:43,034][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:23:43,363][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:23:43,691][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:23:44,021][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:23:44,355][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:23:44,681][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:23:45,011][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:23:45,342][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:23:45,676][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:23:46,006][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:23:46,337][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:23:46,671][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:23:47,001][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:23:47,330][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:23:47,656][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:23:47,987][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:23:48,318][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:23:48,646][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:23:48,974][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:23:49,302][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:23:49,629][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:23:49,955][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:23:50,704][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:23:51,410][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:23:51,412][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:23:51,413][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:23:52,374][__main__][INFO] - Iteration 223 took 22s (36.23% Gen, 59.46% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 16m 32s. Estimated total time: 18h 36m 35s. Time estimates for 10 more iterations: 3m 43s, 100 more iterations: 37m 13s, 500 more iterations: 3h 6m 5s.
[2025-11-13 09:23:52,376][__main__][INFO] - Starting iteration 223.
[2025-11-13 09:23:52,378][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:23:52,379][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:24:00,300][__main__][INFO] - Number of regex retries in iteration 223: 0
[2025-11-13 09:24:00,301][__main__][INFO] - agents played in iteration 223 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:24:00,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:00,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:00,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:00,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:00,884][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:24:00,884][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:24:01,608][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:24:01,906][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:24:02,232][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:24:02,558][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:24:02,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:24:03,212][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:24:03,540][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:24:03,866][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:24:04,197][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:24:04,527][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:24:04,855][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:24:05,182][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:24:05,516][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:24:05,846][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:24:06,177][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:24:06,505][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:24:06,833][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:24:07,159][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:24:07,491][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:24:07,817][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:24:08,145][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:24:08,479][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:24:08,809][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:24:09,140][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:24:09,468][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:24:09,800][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:24:10,132][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:24:10,459][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:24:10,788][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:24:11,116][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:24:11,455][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:24:11,782][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:24:12,110][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:24:12,867][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:24:13,579][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:13,580][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:13,582][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:24:14,520][__main__][INFO] - Iteration 224 took 22s (35.78% Gen, 59.98% Train). Generation: 7s, Training: 13s. Estimated remaining time: 17h 6m 41s. Estimated total time: 18h 27m 7s. Time estimates for 10 more iterations: 3m 41s, 100 more iterations: 36m 54s, 500 more iterations: 3h 4m 31s.
[2025-11-13 09:24:14,522][__main__][INFO] - Starting iteration 224.
[2025-11-13 09:24:14,525][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:24:14,525][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:24:22,752][__main__][INFO] - Number of regex retries in iteration 224: 0
[2025-11-13 09:24:22,753][__main__][INFO] - agents played in iteration 224 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:24:23,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:23,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:23,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:23,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:23,323][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:24:23,323][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:24:24,046][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:24:24,344][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:24:24,670][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:24:24,999][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:24:25,327][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:24:25,654][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:24:25,996][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:24:26,327][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:24:26,658][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:24:26,986][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:24:27,319][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:24:27,647][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:24:27,974][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:24:28,309][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:24:28,631][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:24:28,957][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:24:29,284][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:24:29,613][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:24:29,938][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:24:30,267][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:24:30,594][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:24:30,922][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:24:31,248][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:24:31,576][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:24:31,910][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:24:32,240][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:24:32,569][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:24:32,896][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:24:33,224][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:24:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:24:33,879][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:24:34,207][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:24:34,541][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:24:35,293][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:24:36,002][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:36,004][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:36,005][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:24:36,927][__main__][INFO] - Iteration 225 took 22s (36.72% Gen, 59.15% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 19m 21s. Estimated total time: 18h 40m 9s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 20s, 500 more iterations: 3h 6m 41s.
[2025-11-13 09:24:36,929][__main__][INFO] - Starting iteration 225.
[2025-11-13 09:24:36,932][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:24:36,932][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:24:45,617][__main__][INFO] - Number of regex retries in iteration 225: 0
[2025-11-13 09:24:45,618][__main__][INFO] - agents played in iteration 225 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:24:46,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:46,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:46,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:46,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:24:46,182][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:24:46,183][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:24:46,917][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:24:47,214][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:24:47,540][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:24:47,866][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:24:48,196][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:24:48,524][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:24:48,857][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:24:49,187][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:24:49,522][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:24:49,848][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:24:50,179][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:24:50,514][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:24:50,849][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:24:51,178][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:24:51,513][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:24:51,847][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:24:52,185][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:24:52,519][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:24:52,852][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:24:53,185][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:24:53,517][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:24:53,843][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:24:54,169][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:24:54,496][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:24:54,825][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:24:55,152][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:24:55,480][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:24:55,811][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:24:56,136][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:24:56,464][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:24:56,792][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:24:57,125][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:24:57,451][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:24:58,217][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:24:58,960][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:24:58,961][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:24:58,963][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:25:00,098][__main__][INFO] - Iteration 226 took 23s (37.49% Gen, 57.60% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 57m 11s. Estimated total time: 19h 18m 22s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 3s.
[2025-11-13 09:25:00,100][__main__][INFO] - Starting iteration 226.
[2025-11-13 09:25:00,103][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:25:00,103][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:08,657][__main__][INFO] - Number of regex retries in iteration 226: 0
[2025-11-13 09:25:08,658][__main__][INFO] - agents played in iteration 226 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:25:09,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:09,140][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:09,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:09,232][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:09,233][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:09,233][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:09,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:25:10,258][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:25:10,585][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:25:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:25:11,242][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:25:11,570][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:25:11,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:25:12,226][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:25:12,553][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:25:12,880][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:25:13,210][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:25:13,540][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:25:13,870][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:25:14,199][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:25:14,531][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:25:14,856][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:25:15,182][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:25:15,511][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:25:15,838][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:25:16,164][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:25:16,492][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:25:16,819][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:25:17,146][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:25:17,475][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:25:17,801][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:25:18,129][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:25:18,457][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:25:18,785][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:25:19,112][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:25:19,440][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:25:19,768][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:25:20,094][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:25:20,421][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:25:21,198][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:25:21,902][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:25:21,903][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:25:21,905][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:25:22,905][__main__][INFO] - Iteration 227 took 22s (37.51% Gen, 58.10% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 38m 33s. Estimated total time: 19h 0m 7s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 0s, 500 more iterations: 3h 10m 1s.
[2025-11-13 09:25:22,907][__main__][INFO] - Starting iteration 227.
[2025-11-13 09:25:22,910][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:25:22,910][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:31,979][__main__][INFO] - Number of regex retries in iteration 227: 0
[2025-11-13 09:25:31,980][__main__][INFO] - agents played in iteration 227 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:25:32,423][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:32,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:32,502][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:32,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:32,542][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:32,543][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:33,282][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:25:33,580][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:25:33,907][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:25:34,234][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:25:34,564][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:25:34,894][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:25:35,222][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:25:35,550][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:25:35,877][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:25:36,205][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:25:36,531][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:25:36,858][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:25:37,186][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:25:37,520][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:25:37,847][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:25:38,174][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:25:38,501][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:25:38,831][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:25:39,158][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:25:39,489][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:25:39,817][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:25:40,143][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:25:40,472][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:25:40,801][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:25:41,130][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:25:41,456][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:25:41,787][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:25:42,114][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:25:42,443][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:25:42,771][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:25:43,101][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:25:43,432][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:25:43,758][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:25:44,536][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:25:45,259][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:25:45,261][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:25:45,262][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:25:46,294][__main__][INFO] - Iteration 228 took 23s (38.78% Gen, 56.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 7m 16s. Estimated total time: 19h 29m 14s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 52s.
[2025-11-13 09:25:46,296][__main__][INFO] - Starting iteration 228.
[2025-11-13 09:25:46,299][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:25:46,299][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:25:55,357][__main__][INFO] - Number of regex retries in iteration 228: 0
[2025-11-13 09:25:55,358][__main__][INFO] - agents played in iteration 228 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:25:55,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:55,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:55,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:55,935][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:25:55,936][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:25:55,937][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:25:56,672][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:25:56,969][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:25:57,297][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:25:57,630][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:25:57,957][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:25:58,287][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:25:58,617][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:25:58,946][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:25:59,272][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:25:59,599][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:25:59,925][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:26:00,252][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:26:00,579][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:26:00,919][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:26:01,249][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:26:01,574][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:26:01,900][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:26:02,229][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:26:02,556][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:26:02,882][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:26:03,209][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:26:03,538][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:26:03,865][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:26:04,191][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:26:04,517][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:26:04,846][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:26:05,176][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:26:05,504][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:26:05,837][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:26:06,162][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:26:06,490][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:26:06,820][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:26:07,149][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:26:07,902][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:26:08,627][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:08,629][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:08,631][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:09,548][__main__][INFO] - Iteration 229 took 23s (38.96% Gen, 57.09% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 0m 9s. Estimated total time: 19h 22m 30s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 45s, 500 more iterations: 3h 13m 45s.
[2025-11-13 09:26:09,550][__main__][INFO] - Starting iteration 229.
[2025-11-13 09:26:09,553][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:26:09,553][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:26:18,312][__main__][INFO] - Number of regex retries in iteration 229: 0
[2025-11-13 09:26:18,313][__main__][INFO] - agents played in iteration 229 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:26:18,762][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:18,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:18,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:18,898][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:18,899][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:26:18,899][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:26:19,631][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:26:19,928][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:26:20,254][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:26:20,581][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:26:20,909][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:26:21,237][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:26:21,567][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:26:21,894][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:26:22,221][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:26:22,547][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:26:22,873][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:26:23,200][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:26:23,526][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:26:23,852][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:26:24,178][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:26:24,505][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:26:24,832][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:26:25,159][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:26:25,488][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:26:25,817][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:26:26,145][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:26:26,475][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:26:26,809][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:26:27,143][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:26:27,473][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:26:27,799][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:26:28,128][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:26:28,455][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:26:28,782][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:26:29,114][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:26:29,442][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:26:29,770][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:26:30,103][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:26:30,865][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:26:31,580][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:31,582][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:31,583][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:32,822][__main__][INFO] - Iteration 230 took 23s (37.64% Gen, 57.03% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 0m 47s. Estimated total time: 19h 23m 31s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 47s, 500 more iterations: 3h 13m 55s.
[2025-11-13 09:26:32,824][__main__][INFO] - Starting iteration 230.
[2025-11-13 09:26:32,827][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 22 and human policies 1.
[2025-11-13 09:26:32,828][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:26:41,453][__main__][INFO] - Number of regex retries in iteration 230: 0
[2025-11-13 09:26:41,453][__main__][INFO] - agents played in iteration 230 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:26:41,908][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:41,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:41,991][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:42,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:26:42,031][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:26:42,031][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:26:42,822][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:26:43,121][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:26:43,452][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:26:43,787][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:26:44,128][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:26:44,456][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:26:44,784][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:26:45,120][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:26:45,446][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:26:45,780][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:26:46,111][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:26:46,447][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:26:46,770][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:26:47,096][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:26:47,422][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:26:47,750][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:26:48,078][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:26:48,404][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:26:48,730][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:26:49,061][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:26:49,388][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:26:49,714][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:26:50,040][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:26:50,366][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:26:50,694][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:26:51,021][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:26:51,349][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:26:51,677][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:26:52,006][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:26:52,337][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:26:52,664][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:26:52,994][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:26:53,324][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:26:54,090][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:26:54,827][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:26:54,828][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:26:54,830][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:26:56,683][__main__][INFO] - Iteration 231 took 23s (36.16% Gen, 56.07% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 29m 41s. Estimated total time: 19h 52m 49s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 45s, 500 more iterations: 3h 18m 48s.
[2025-11-13 09:26:56,685][__main__][INFO] - Starting iteration 231.
[2025-11-13 09:26:56,687][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:26:56,688][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:05,926][__main__][INFO] - Number of regex retries in iteration 231: 0
[2025-11-13 09:27:05,927][__main__][INFO] - agents played in iteration 231 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:27:06,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:06,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:06,471][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:06,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:06,512][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:06,512][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:07,251][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:27:07,549][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:27:07,876][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:27:08,206][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:27:08,532][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:27:08,858][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:27:09,185][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:27:09,511][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:27:09,837][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:27:10,165][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:27:10,490][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:27:10,818][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:27:11,145][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:27:11,483][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:27:11,813][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:27:12,143][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:27:12,470][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:27:12,804][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:27:13,131][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:27:13,457][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:27:13,783][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:27:14,122][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:27:14,449][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:27:14,776][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:27:15,105][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:27:15,435][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:27:15,770][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:27:16,100][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:27:16,433][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:27:16,756][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:27:17,084][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:27:17,413][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:27:17,742][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:27:18,519][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:27:19,252][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:27:19,253][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:27:19,255][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:27:20,209][__main__][INFO] - Iteration 232 took 23s (39.27% Gen, 56.66% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 12m 35s. Estimated total time: 19h 36m 7s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 1s.
[2025-11-13 09:27:20,211][__main__][INFO] - Starting iteration 232.
[2025-11-13 09:27:20,214][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:27:20,215][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:29,315][__main__][INFO] - Number of regex retries in iteration 232: 0
[2025-11-13 09:27:29,316][__main__][INFO] - agents played in iteration 232 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:27:29,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:29,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:29,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:29,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:29,875][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:29,875][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:30,629][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:27:30,927][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:27:31,253][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:27:31,581][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:27:31,911][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:27:32,238][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:27:32,566][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:27:32,892][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:27:33,221][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:27:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:27:33,878][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:27:34,205][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:27:34,537][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:27:34,864][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:27:35,193][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:27:35,523][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:27:35,855][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:27:36,182][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:27:36,511][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:27:36,845][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:27:37,172][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:27:37,501][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:27:37,828][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:27:38,158][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:27:38,487][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:27:38,817][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:27:39,146][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:27:39,473][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:27:39,801][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:27:40,134][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:27:40,462][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:27:40,790][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:27:41,118][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:27:41,873][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:27:42,590][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:27:42,591][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:27:42,593][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:27:43,593][__main__][INFO] - Iteration 233 took 23s (38.93% Gen, 56.79% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 5m 4s. Estimated total time: 19h 28m 59s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 57s, 500 more iterations: 3h 14m 49s.
[2025-11-13 09:27:43,595][__main__][INFO] - Starting iteration 233.
[2025-11-13 09:27:43,598][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:27:43,600][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:27:52,674][__main__][INFO] - Number of regex retries in iteration 233: 0
[2025-11-13 09:27:52,674][__main__][INFO] - agents played in iteration 233 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:27:53,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:53,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:53,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:53,268][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:27:53,268][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:27:53,269][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:27:54,003][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:27:54,301][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:27:54,630][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:27:54,956][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:27:55,286][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:27:55,612][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:27:55,939][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:27:56,267][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:27:56,593][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:27:56,920][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:27:57,247][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:27:57,573][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:27:57,900][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:27:58,227][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:27:58,554][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:27:58,879][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:27:59,209][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:27:59,536][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:27:59,863][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:28:00,191][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:28:00,518][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:28:00,845][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:28:01,180][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:28:01,509][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:28:01,839][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:28:02,164][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:28:02,492][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:28:02,822][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:28:03,155][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:28:03,483][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:28:03,812][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:28:04,139][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:28:04,468][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:28:05,217][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:28:05,933][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:05,935][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:05,937][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:06,861][__main__][INFO] - Iteration 234 took 23s (39.01% Gen, 57.01% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 58m 54s. Estimated total time: 19h 23m 12s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 46s, 500 more iterations: 3h 13m 52s.
[2025-11-13 09:28:06,863][__main__][INFO] - Starting iteration 234.
[2025-11-13 09:28:06,866][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:28:06,866][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:28:15,747][__main__][INFO] - Number of regex retries in iteration 234: 0
[2025-11-13 09:28:15,748][__main__][INFO] - agents played in iteration 234 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:28:16,212][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:16,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:16,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:16,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:16,333][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:28:16,333][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:28:17,073][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:28:17,370][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:28:17,703][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:28:18,036][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:28:18,365][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:28:18,693][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:28:19,020][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:28:19,348][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:28:19,675][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:28:20,004][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:28:20,332][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:28:20,661][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:28:20,990][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:28:21,319][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:28:21,646][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:28:21,974][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:28:22,300][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:28:22,627][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:28:22,954][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:28:23,281][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:28:23,609][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:28:23,936][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:28:24,264][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:28:24,592][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:28:24,922][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:28:25,252][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:28:25,583][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:28:25,909][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:28:26,237][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:28:26,564][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:28:26,891][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:28:27,219][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:28:27,553][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:28:28,288][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:28:29,013][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:29,014][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:29,016][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:29,948][__main__][INFO] - Iteration 235 took 23s (38.47% Gen, 57.48% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 49m 27s. Estimated total time: 19h 14m 8s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 21s.
[2025-11-13 09:28:29,950][__main__][INFO] - Starting iteration 235.
[2025-11-13 09:28:29,953][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:28:29,953][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:28:39,469][__main__][INFO] - Number of regex retries in iteration 235: 0
[2025-11-13 09:28:39,470][__main__][INFO] - agents played in iteration 235 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:28:39,925][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:39,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:40,006][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:40,046][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:28:40,046][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:28:40,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:28:40,809][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:28:41,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:28:41,434][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:28:41,764][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:28:42,091][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:28:42,420][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:28:42,746][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:28:43,074][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:28:43,407][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:28:43,747][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:28:44,076][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:28:44,403][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:28:44,730][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:28:45,056][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:28:45,390][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:28:45,716][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:28:46,043][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:28:46,372][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:28:46,700][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:28:47,029][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:28:47,356][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:28:47,683][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:28:48,010][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:28:48,337][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:28:48,667][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:28:48,992][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:28:49,320][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:28:49,647][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:28:49,975][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:28:50,302][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:28:50,629][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:28:50,957][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:28:51,285][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:28:52,036][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:28:52,771][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:28:52,773][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:28:52,774][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:28:53,794][__main__][INFO] - Iteration 236 took 23s (39.91% Gen, 55.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 27m 2s. Estimated total time: 19h 52m 7s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 44s, 500 more iterations: 3h 18m 41s.
[2025-11-13 09:28:53,797][__main__][INFO] - Starting iteration 236.
[2025-11-13 09:28:53,800][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:28:53,801][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:29:03,084][__main__][INFO] - Number of regex retries in iteration 236: 0
[2025-11-13 09:29:03,084][__main__][INFO] - agents played in iteration 236 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:29:03,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:03,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:03,621][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:03,662][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:03,663][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:29:03,663][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:29:04,443][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:29:04,740][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:29:05,067][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:29:05,393][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:29:05,720][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:29:06,053][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:29:06,379][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:29:06,707][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:29:07,035][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:29:07,366][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:29:07,694][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:29:08,020][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:29:08,353][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:29:08,680][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:29:09,007][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:29:09,332][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:29:09,659][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:29:09,988][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:29:10,316][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:29:10,645][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:29:10,975][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:29:11,302][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:29:11,629][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:29:11,957][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:29:12,285][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:29:12,613][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:29:12,941][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:29:13,269][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:29:13,596][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:29:13,929][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:29:14,257][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:29:14,585][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:29:14,911][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:29:15,678][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:29:16,440][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:29:16,442][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:29:16,444][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:29:17,734][__main__][INFO] - Iteration 237 took 23s (38.78% Gen, 55.81% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 31m 17s. Estimated total time: 19h 56m 46s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 53s, 500 more iterations: 3h 19m 27s.
[2025-11-13 09:29:17,737][__main__][INFO] - Starting iteration 237.
[2025-11-13 09:29:17,740][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:29:17,741][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:29:27,263][__main__][INFO] - Number of regex retries in iteration 237: 0
[2025-11-13 09:29:27,264][__main__][INFO] - agents played in iteration 237 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:29:27,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:27,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:27,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:27,843][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:27,844][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:29:27,844][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:29:28,630][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:29:28,928][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:29:29,254][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:29:29,580][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:29:29,908][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:29:30,237][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:29:30,570][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:29:30,898][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:29:31,225][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:29:31,554][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:29:31,887][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:29:32,218][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:29:32,552][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:29:32,886][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:29:33,216][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:29:33,543][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:29:33,870][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:29:34,198][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:29:34,526][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:29:34,854][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:29:35,182][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:29:35,508][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:29:35,835][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:29:36,168][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:29:36,493][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:29:36,820][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:29:37,151][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:29:37,478][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:29:37,805][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:29:38,133][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:29:38,462][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:29:38,790][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:29:39,118][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:29:39,861][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:29:40,632][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:29:40,635][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:29:40,637][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:29:41,647][__main__][INFO] - Iteration 238 took 23s (39.83% Gen, 55.94% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 29m 29s. Estimated total time: 19h 55m 22s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 13s.
[2025-11-13 09:29:41,649][__main__][INFO] - Starting iteration 238.
[2025-11-13 09:29:41,654][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:29:41,656][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:29:50,607][__main__][INFO] - Number of regex retries in iteration 238: 0
[2025-11-13 09:29:50,608][__main__][INFO] - agents played in iteration 238 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:29:51,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:51,117][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:51,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:51,196][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:29:51,197][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:29:51,197][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:29:51,957][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:29:52,259][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:29:52,582][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:29:52,908][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:29:53,234][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:29:53,566][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:29:53,885][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:29:54,211][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:29:54,537][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:29:54,871][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:29:55,195][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:29:55,523][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:29:55,850][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:29:56,178][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:29:56,507][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:29:56,837][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:29:57,165][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:29:57,492][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:29:57,821][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:29:58,149][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:29:58,476][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:29:58,802][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:29:59,134][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:29:59,462][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:29:59,789][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:30:00,116][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:30:00,449][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:30:00,778][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:30:01,107][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:30:01,435][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:30:01,764][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:30:02,097][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:30:02,431][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:30:03,187][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:30:03,943][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:30:03,946][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:30:03,948][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:30:05,031][__main__][INFO] - Iteration 239 took 23s (38.28% Gen, 57.06% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 2m 43s. Estimated total time: 19h 28m 59s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 57s, 500 more iterations: 3h 14m 49s.
[2025-11-13 09:30:05,033][__main__][INFO] - Starting iteration 239.
[2025-11-13 09:30:05,037][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:30:05,037][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:30:13,832][__main__][INFO] - Number of regex retries in iteration 239: 0
[2025-11-13 09:30:13,833][__main__][INFO] - agents played in iteration 239 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:30:14,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:14,329][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:14,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:14,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:14,408][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:30:14,409][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:30:15,186][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:30:15,483][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:30:15,815][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:30:16,141][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:30:16,475][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:30:16,801][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:30:17,127][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:30:17,453][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:30:17,783][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:30:18,114][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:30:18,442][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:30:18,771][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:30:19,094][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:30:19,421][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:30:19,748][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:30:20,081][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:30:20,408][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:30:20,739][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:30:21,067][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:30:21,394][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:30:21,722][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:30:22,049][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:30:22,376][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:30:22,703][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:30:23,032][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:30:23,362][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:30:23,690][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:30:24,018][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:30:24,355][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:30:24,688][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:30:25,022][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:30:25,349][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:30:25,676][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:30:26,414][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:30:27,204][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:30:27,206][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:30:27,207][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:30:28,300][__main__][INFO] - Iteration 240 took 23s (37.81% Gen, 57.49% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 56m 32s. Estimated total time: 19h 23m 11s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 46s, 500 more iterations: 3h 13m 51s.
[2025-11-13 09:30:28,302][__main__][INFO] - Starting iteration 240.
[2025-11-13 09:30:28,306][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 23 and human policies 1.
[2025-11-13 09:30:28,307][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:30:37,689][__main__][INFO] - Number of regex retries in iteration 240: 0
[2025-11-13 09:30:37,690][__main__][INFO] - agents played in iteration 240 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:30:38,146][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:38,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:38,226][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:38,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:30:38,266][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:30:38,267][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:30:39,047][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:30:39,344][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:30:39,672][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:30:39,999][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:30:40,326][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:30:40,654][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:30:40,980][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:30:41,308][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:30:41,639][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:30:41,967][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:30:42,297][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:30:42,624][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:30:42,952][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:30:43,279][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:30:43,606][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:30:43,933][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:30:44,262][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:30:44,590][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:30:44,917][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:30:45,245][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:30:45,581][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:30:45,904][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:30:46,232][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:30:46,559][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:30:46,887][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:30:47,216][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:30:47,544][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:30:47,871][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:30:48,199][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:30:48,538][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:30:48,865][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:30:49,192][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:30:49,519][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:30:50,262][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:30:51,028][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:30:51,030][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:30:51,031][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:30:53,368][__main__][INFO] - Iteration 241 took 25s (37.43% Gen, 53.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 26m 6s. Estimated total time: 20h 53m 11s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 46s, 500 more iterations: 3h 28m 51s.
[2025-11-13 09:30:53,370][__main__][INFO] - Starting iteration 241.
[2025-11-13 09:30:53,373][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:30:53,374][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:31:02,532][__main__][INFO] - Number of regex retries in iteration 241: 0
[2025-11-13 09:31:02,533][__main__][INFO] - agents played in iteration 241 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:31:03,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,126][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:03,127][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:31:03,127][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:31:03,879][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:31:04,183][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:31:04,509][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:31:04,835][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:31:05,164][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:31:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:31:05,820][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:31:06,148][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:31:06,476][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:31:06,804][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:31:07,132][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:31:07,460][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:31:07,787][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:31:08,114][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:31:08,441][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:31:08,768][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:31:09,096][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:31:09,424][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:31:09,752][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:31:10,078][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:31:10,404][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:31:10,731][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:31:11,064][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:31:11,395][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:31:11,727][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:31:12,061][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:31:12,390][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:31:12,718][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:31:13,051][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:31:13,382][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:31:13,711][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:31:14,039][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:31:14,367][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:31:15,136][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:31:15,919][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:31:15,920][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:31:15,922][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:31:16,943][__main__][INFO] - Iteration 242 took 23s (38.85% Gen, 56.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 11m 5s. Estimated total time: 19h 38m 34s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 17s, 500 more iterations: 3h 16m 25s.
[2025-11-13 09:31:16,945][__main__][INFO] - Starting iteration 242.
[2025-11-13 09:31:16,948][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:31:16,949][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:31:26,512][__main__][INFO] - Number of regex retries in iteration 242: 0
[2025-11-13 09:31:26,513][__main__][INFO] - agents played in iteration 242 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:31:26,978][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:27,018][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:27,058][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:27,098][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:27,099][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:31:27,099][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:31:27,859][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:31:28,156][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:31:28,483][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:31:28,811][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:31:29,152][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:31:29,479][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:31:29,808][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:31:30,135][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:31:30,477][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:31:30,804][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:31:31,132][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:31:31,460][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:31:31,790][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:31:32,117][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:31:32,443][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:31:32,772][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:31:33,100][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:31:33,433][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:31:33,761][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:31:34,089][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:31:34,419][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:31:34,750][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:31:35,078][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:31:35,405][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:31:35,733][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:31:36,062][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:31:36,391][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:31:36,718][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:31:37,045][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:31:37,373][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:31:37,702][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:31:38,029][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:31:38,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:31:39,145][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:31:39,916][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:31:39,918][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:31:39,919][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:31:40,911][__main__][INFO] - Iteration 243 took 23s (39.91% Gen, 55.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 30m 20s. Estimated total time: 19h 58m 12s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 56s, 500 more iterations: 3h 19m 42s.
[2025-11-13 09:31:40,913][__main__][INFO] - Starting iteration 243.
[2025-11-13 09:31:40,916][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:31:40,916][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:31:50,150][__main__][INFO] - Number of regex retries in iteration 243: 0
[2025-11-13 09:31:50,151][__main__][INFO] - agents played in iteration 243 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:31:50,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:50,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:50,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:50,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:31:50,747][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:31:50,747][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:31:51,549][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:31:51,847][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:31:52,176][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:31:52,506][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:31:52,833][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:31:53,159][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:31:53,486][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:31:53,814][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:31:54,142][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:31:54,469][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:31:54,797][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:31:55,125][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:31:55,457][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:31:55,784][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:31:56,111][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:31:56,439][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:31:56,767][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:31:57,094][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:31:57,425][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:31:57,753][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:31:58,080][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:31:58,407][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:31:58,734][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:31:59,063][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:31:59,391][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:31:59,717][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:32:00,044][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:32:00,372][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:32:00,705][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:32:01,028][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:32:01,355][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:32:01,683][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:32:02,012][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:32:02,771][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:32:03,551][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:03,553][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:03,554][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:32:04,547][__main__][INFO] - Iteration 244 took 23s (39.08% Gen, 56.72% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 13m 19s. Estimated total time: 19h 41m 35s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 23s, 500 more iterations: 3h 16m 55s.
[2025-11-13 09:32:04,549][__main__][INFO] - Starting iteration 244.
[2025-11-13 09:32:04,553][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:32:04,554][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:32:14,360][__main__][INFO] - Number of regex retries in iteration 244: 0
[2025-11-13 09:32:14,360][__main__][INFO] - agents played in iteration 244 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:32:14,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:14,875][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:14,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:14,956][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:14,957][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:32:14,957][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:32:15,758][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:32:16,057][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:32:16,385][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:32:16,711][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:32:17,038][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:32:17,365][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:32:17,692][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:32:18,020][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:32:18,348][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:32:18,676][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:32:19,002][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:32:19,331][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:32:19,658][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:32:19,986][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:32:20,314][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:32:20,642][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:32:20,974][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:32:21,304][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:32:21,634][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:32:21,963][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:32:22,303][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:32:22,629][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:32:22,963][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:32:23,293][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:32:23,626][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:32:23,956][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:32:24,282][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:32:24,615][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:32:24,938][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:32:25,265][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:32:25,592][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:32:25,922][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:32:26,248][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:32:27,020][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:32:27,788][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:27,789][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:27,791][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:32:28,891][__main__][INFO] - Iteration 245 took 24s (40.29% Gen, 55.18% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 48m 18s. Estimated total time: 20h 16m 58s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 33s, 500 more iterations: 3h 22m 49s.
[2025-11-13 09:32:28,893][__main__][INFO] - Starting iteration 245.
[2025-11-13 09:32:28,897][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:32:28,898][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:32:38,545][__main__][INFO] - Number of regex retries in iteration 245: 0
[2025-11-13 09:32:38,546][__main__][INFO] - agents played in iteration 245 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:32:39,024][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:39,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:39,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:39,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:32:39,162][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:32:39,162][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:32:39,928][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:32:40,226][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:32:40,553][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:32:40,880][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:32:41,207][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:32:41,533][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:32:41,861][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:32:42,186][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:32:42,514][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:32:42,841][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:32:43,167][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:32:43,495][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:32:43,822][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:32:44,154][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:32:44,485][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:32:44,820][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:32:45,146][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:32:45,473][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:32:45,805][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:32:46,132][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:32:46,459][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:32:46,788][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:32:47,115][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:32:47,446][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:32:47,773][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:32:48,102][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:32:48,432][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:32:48,755][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:32:49,084][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:32:49,412][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:32:49,744][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:32:50,071][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:32:50,399][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:32:51,151][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:32:51,913][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:32:51,914][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:32:51,916][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:32:52,937][__main__][INFO] - Iteration 246 took 24s (40.13% Gen, 55.61% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 32m 59s. Estimated total time: 20h 2m 4s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 4s, 500 more iterations: 3h 20m 20s.
[2025-11-13 09:32:52,939][__main__][INFO] - Starting iteration 246.
[2025-11-13 09:32:52,943][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:32:52,943][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:02,362][__main__][INFO] - Number of regex retries in iteration 246: 0
[2025-11-13 09:33:02,362][__main__][INFO] - agents played in iteration 246 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:33:02,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:02,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:02,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:02,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:02,961][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:02,963][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:03,754][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:33:04,052][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:33:04,379][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:33:04,705][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:33:05,031][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:33:05,359][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:33:05,687][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:33:06,013][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:33:06,339][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:33:06,668][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:33:07,000][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:33:07,329][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:33:07,655][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:33:07,983][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:33:08,311][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:33:08,639][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:33:08,967][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:33:09,295][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:33:09,629][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:33:09,951][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:33:10,278][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:33:10,605][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:33:10,932][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:33:11,260][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:33:11,588][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:33:11,917][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:33:12,243][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:33:12,570][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:33:12,896][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:33:13,222][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:33:13,549][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:33:13,876][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:33:14,206][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:33:14,973][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:33:15,721][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:33:15,723][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:33:15,724][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:33:16,878][__main__][INFO] - Iteration 247 took 23s (39.35% Gen, 55.82% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 27m 21s. Estimated total time: 19h 56m 49s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 53s, 500 more iterations: 3h 19m 28s.
[2025-11-13 09:33:16,880][__main__][INFO] - Starting iteration 247.
[2025-11-13 09:33:16,883][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:33:16,884][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:25,817][__main__][INFO] - Number of regex retries in iteration 247: 0
[2025-11-13 09:33:25,817][__main__][INFO] - agents played in iteration 247 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:33:26,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:26,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:26,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:26,400][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:26,401][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:26,401][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:27,192][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:33:27,490][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:33:27,821][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:33:28,144][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:33:28,470][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:33:28,798][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:33:29,136][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:33:29,457][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:33:29,784][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:33:30,110][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:33:30,439][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:33:30,769][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:33:31,104][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:33:31,431][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:33:31,760][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:33:32,090][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:33:32,416][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:33:32,743][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:33:33,070][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:33:33,403][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:33:33,730][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:33:34,059][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:33:34,385][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:33:34,724][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:33:35,050][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:33:35,377][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:33:35,704][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:33:36,035][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:33:36,362][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:33:36,689][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:33:37,017][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:33:37,343][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:33:37,670][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:33:38,417][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:33:39,183][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:33:39,185][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:33:39,187][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:33:40,216][__main__][INFO] - Iteration 248 took 23s (38.28% Gen, 57.29% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 56m 51s. Estimated total time: 19h 26m 42s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 53s, 500 more iterations: 3h 14m 27s.
[2025-11-13 09:33:40,218][__main__][INFO] - Starting iteration 248.
[2025-11-13 09:33:40,222][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:33:40,222][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:33:49,562][__main__][INFO] - Number of regex retries in iteration 248: 0
[2025-11-13 09:33:49,563][__main__][INFO] - agents played in iteration 248 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:33:50,023][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:50,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:50,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:50,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:33:50,143][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:33:50,144][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:33:50,923][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:33:51,223][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:33:51,549][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:33:51,875][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:33:52,215][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:33:52,544][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:33:52,870][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:33:53,196][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:33:53,528][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:33:53,857][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:33:54,188][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:33:54,518][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:33:54,843][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:33:55,169][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:33:55,495][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:33:55,826][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:33:56,147][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:33:56,475][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:33:56,801][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:33:57,133][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:33:57,455][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:33:57,781][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:33:58,107][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:33:58,437][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:33:58,759][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:33:59,086][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:33:59,414][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:33:59,743][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:34:00,081][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:34:00,412][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:34:00,745][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:34:01,075][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:34:01,402][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:34:02,118][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:34:02,884][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:34:02,886][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:34:02,887][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:34:03,921][__main__][INFO] - Iteration 249 took 23s (39.41% Gen, 56.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 14m 45s. Estimated total time: 19h 45m 0s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 30s.
[2025-11-13 09:34:03,923][__main__][INFO] - Starting iteration 249.
[2025-11-13 09:34:03,927][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:34:03,927][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:34:13,692][__main__][INFO] - Number of regex retries in iteration 249: 0
[2025-11-13 09:34:13,693][__main__][INFO] - agents played in iteration 249 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:34:14,167][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:14,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:14,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:14,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:14,290][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:34:14,290][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:34:15,073][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:34:15,370][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:34:15,711][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:34:16,039][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:34:16,369][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:34:16,697][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:34:17,032][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:34:17,367][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:34:17,695][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:34:18,021][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:34:18,347][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:34:18,677][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:34:19,003][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:34:19,333][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:34:19,655][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:34:19,982][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:34:20,314][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:34:20,647][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:34:20,977][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:34:21,306][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:34:21,633][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:34:21,962][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:34:22,299][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:34:22,627][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:34:22,953][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:34:23,286][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:34:23,615][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:34:23,945][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:34:24,272][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:34:24,597][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:34:24,924][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:34:25,252][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:34:25,579][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:34:26,334][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:34:27,104][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:34:27,105][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:34:27,107][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:34:28,379][__main__][INFO] - Iteration 250 took 24s (39.94% Gen, 54.86% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 52m 0s. Estimated total time: 20h 22m 40s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 45s, 500 more iterations: 3h 23m 46s.
[2025-11-13 09:34:28,381][__main__][INFO] - Starting iteration 250.
[2025-11-13 09:34:28,384][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 24 and human policies 1.
[2025-11-13 09:34:28,385][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:34:37,942][__main__][INFO] - Number of regex retries in iteration 250: 0
[2025-11-13 09:34:37,943][__main__][INFO] - agents played in iteration 250 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:34:38,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:38,434][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:38,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:38,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:34:38,515][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:34:38,515][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:34:39,310][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:34:39,609][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:34:39,939][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:34:40,269][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:34:40,602][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:34:40,928][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:34:41,255][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:34:41,581][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:34:41,921][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:34:42,248][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:34:42,576][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:34:42,901][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:34:43,237][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:34:43,564][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:34:43,889][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:34:44,216][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:34:44,548][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:34:44,882][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:34:45,212][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:34:45,542][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:34:45,870][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:34:46,199][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:34:46,528][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:34:46,857][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:34:47,195][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:34:47,528][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:34:47,857][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:34:48,186][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:34:48,528][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:34:48,854][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:34:49,182][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:34:49,511][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:34:49,845][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:34:50,543][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:34:51,311][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:34:51,312][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:34:51,314][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:34:53,394][__main__][INFO] - Iteration 251 took 25s (38.22% Gen, 53.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 19m 27s. Estimated total time: 20h 50m 32s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 41s, 500 more iterations: 3h 28m 25s.
[2025-11-13 09:34:53,396][__main__][INFO] - Starting iteration 251.
[2025-11-13 09:34:53,399][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:34:53,400][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:03,246][__main__][INFO] - Number of regex retries in iteration 251: 0
[2025-11-13 09:35:03,247][__main__][INFO] - agents played in iteration 251 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:35:03,694][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:03,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:03,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:03,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:03,816][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:03,817][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:04,621][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:35:04,918][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:35:05,252][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:35:05,579][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:35:05,920][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:35:06,253][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:35:06,581][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:35:06,908][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:35:07,240][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:35:07,566][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:35:07,893][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:35:08,220][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:35:08,549][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:35:08,878][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:35:09,207][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:35:09,538][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:35:09,861][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:35:10,188][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:35:10,514][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:35:10,845][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:35:11,172][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:35:11,499][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:35:11,826][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:35:12,154][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:35:12,480][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:35:12,810][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:35:13,140][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:35:13,466][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:35:13,793][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:35:14,120][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:35:14,447][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:35:14,774][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:35:15,106][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:35:15,841][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:35:16,625][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:35:16,627][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:35:16,628][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:35:17,660][__main__][INFO] - Iteration 252 took 24s (40.59% Gen, 55.16% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 41m 37s. Estimated total time: 20h 13m 5s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 26s, 500 more iterations: 3h 22m 10s.
[2025-11-13 09:35:17,662][__main__][INFO] - Starting iteration 252.
[2025-11-13 09:35:17,666][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:35:17,666][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:27,899][__main__][INFO] - Number of regex retries in iteration 252: 0
[2025-11-13 09:35:27,900][__main__][INFO] - agents played in iteration 252 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:35:28,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:28,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:28,445][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:28,485][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:28,485][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:28,486][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:29,255][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:35:29,552][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:35:29,882][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:35:30,212][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:35:30,542][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:35:30,874][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:35:31,200][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:35:31,532][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:35:31,862][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:35:32,192][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:35:32,522][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:35:32,852][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:35:33,178][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:35:33,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:35:33,854][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:35:34,182][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:35:34,508][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:35:34,844][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:35:35,178][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:35:35,511][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:35:35,845][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:35:36,173][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:35:36,500][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:35:36,826][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:35:37,152][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:35:37,481][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:35:37,810][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:35:38,139][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:35:38,465][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:35:38,792][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:35:39,119][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:35:39,447][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:35:39,775][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:35:40,518][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:35:41,286][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:35:41,287][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:35:41,289][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:35:42,256][__main__][INFO] - Iteration 253 took 24s (41.62% Gen, 54.45% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 57m 38s. Estimated total time: 20h 29m 32s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 59s, 500 more iterations: 3h 24m 55s.
[2025-11-13 09:35:42,258][__main__][INFO] - Starting iteration 253.
[2025-11-13 09:35:42,261][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:35:42,261][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:35:52,338][__main__][INFO] - Number of regex retries in iteration 253: 0
[2025-11-13 09:35:52,339][__main__][INFO] - agents played in iteration 253 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:35:52,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:52,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:52,877][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:52,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:35:52,917][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:35:52,918][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:35:53,693][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:35:53,989][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:35:54,322][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:35:54,656][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:35:54,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:35:55,320][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:35:55,648][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:35:55,978][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:35:56,305][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:35:56,631][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:35:56,961][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:35:57,284][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:35:57,610][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:35:57,939][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:35:58,266][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:35:58,598][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:35:58,923][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:35:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:35:59,575][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:35:59,903][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:36:00,232][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:36:00,560][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:36:00,885][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:36:01,213][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:36:01,539][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:36:01,868][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:36:02,200][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:36:02,531][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:36:02,859][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:36:03,188][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:36:03,517][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:36:03,844][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:36:04,172][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:36:04,906][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:36:05,683][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:36:05,684][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:36:05,686][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:06,690][__main__][INFO] - Iteration 254 took 24s (41.25% Gen, 54.63% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 49m 13s. Estimated total time: 20h 21m 31s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 43s, 500 more iterations: 3h 23m 35s.
[2025-11-13 09:36:06,692][__main__][INFO] - Starting iteration 254.
[2025-11-13 09:36:06,697][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:36:06,697][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:36:16,957][__main__][INFO] - Number of regex retries in iteration 254: 0
[2025-11-13 09:36:16,958][__main__][INFO] - agents played in iteration 254 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:36:17,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,552][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:17,552][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:36:17,553][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:36:18,327][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:36:18,625][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:36:18,953][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:36:19,278][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:36:19,605][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:36:19,933][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:36:20,258][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:36:20,584][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:36:20,910][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:36:21,235][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:36:21,560][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:36:21,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:36:22,210][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:36:22,537][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:36:22,861][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:36:23,187][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:36:23,515][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:36:23,840][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:36:24,166][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:36:24,492][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:36:24,818][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:36:25,144][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:36:25,469][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:36:25,795][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:36:26,125][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:36:26,450][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:36:26,775][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:36:27,101][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:36:27,429][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:36:27,754][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:36:28,079][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:36:28,406][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:36:28,734][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:36:29,497][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:36:30,277][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:36:30,279][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:36:30,281][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:31,280][__main__][INFO] - Iteration 255 took 24s (41.74% Gen, 54.19% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 56m 31s. Estimated total time: 20h 29m 14s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 58s, 500 more iterations: 3h 24m 52s.
[2025-11-13 09:36:31,283][__main__][INFO] - Starting iteration 255.
[2025-11-13 09:36:31,286][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:36:31,286][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:36:41,391][__main__][INFO] - Number of regex retries in iteration 255: 0
[2025-11-13 09:36:41,392][__main__][INFO] - agents played in iteration 255 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:36:41,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:41,892][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:41,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:41,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:36:41,974][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:36:41,975][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:36:42,768][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:36:43,065][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:36:43,393][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:36:43,721][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:36:44,055][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:36:44,381][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:36:44,715][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:36:45,045][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:36:45,374][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:36:45,704][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:36:46,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:36:46,364][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:36:46,690][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:36:47,017][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:36:47,352][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:36:47,680][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:36:48,006][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:36:48,333][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:36:48,660][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:36:48,987][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:36:49,319][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:36:49,649][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:36:49,978][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:36:50,304][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:36:50,631][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:36:50,964][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:36:51,291][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:36:51,623][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:36:51,952][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:36:52,278][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:36:52,604][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:36:52,931][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:36:53,259][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:36:53,984][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:36:54,730][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:36:54,731][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:36:54,733][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:36:55,747][__main__][INFO] - Iteration 256 took 24s (41.31% Gen, 54.54% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 49m 57s. Estimated total time: 20h 23m 4s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 46s, 500 more iterations: 3h 23m 50s.
[2025-11-13 09:36:55,749][__main__][INFO] - Starting iteration 256.
[2025-11-13 09:36:55,752][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:36:55,753][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:05,715][__main__][INFO] - Number of regex retries in iteration 256: 0
[2025-11-13 09:37:05,716][__main__][INFO] - agents played in iteration 256 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:37:06,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,224][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:06,318][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:06,318][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:07,102][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:37:07,399][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:37:07,728][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:37:08,065][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:37:08,391][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:37:08,721][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:37:09,048][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:37:09,382][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:37:09,711][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:37:10,046][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:37:10,375][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:37:10,709][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:37:11,038][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:37:11,370][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:37:11,699][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:37:12,028][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:37:12,360][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:37:12,690][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:37:13,022][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:37:13,350][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:37:13,676][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:37:14,005][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:37:14,334][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:37:14,660][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:37:14,986][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:37:15,313][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:37:15,640][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:37:15,966][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:37:16,293][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:37:16,623][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:37:16,952][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:37:17,281][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:37:17,608][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:37:18,362][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:37:19,125][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:37:19,126][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:37:19,128][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:37:20,117][__main__][INFO] - Iteration 257 took 24s (40.89% Gen, 55.04% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 44m 46s. Estimated total time: 20h 18m 17s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 36s, 500 more iterations: 3h 23m 2s.
[2025-11-13 09:37:20,120][__main__][INFO] - Starting iteration 257.
[2025-11-13 09:37:20,123][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:37:20,124][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:29,774][__main__][INFO] - Number of regex retries in iteration 257: 0
[2025-11-13 09:37:29,775][__main__][INFO] - agents played in iteration 257 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:37:30,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:30,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:30,334][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:30,382][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:30,383][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:30,384][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:31,201][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:37:31,500][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:37:31,829][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:37:32,156][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:37:32,485][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:37:32,814][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:37:33,143][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:37:33,472][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:37:33,799][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:37:34,125][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:37:34,451][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:37:34,778][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:37:35,105][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:37:35,432][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:37:35,759][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:37:36,089][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:37:36,416][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:37:36,741][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:37:37,068][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:37:37,397][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:37:37,723][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:37:38,049][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:37:38,375][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:37:38,703][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:37:39,028][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:37:39,354][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:37:39,680][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:37:40,007][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:37:40,334][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:37:40,660][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:37:40,986][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:37:41,314][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:37:41,640][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:37:42,384][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:37:43,154][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:37:43,155][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:37:43,157][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:37:44,177][__main__][INFO] - Iteration 258 took 24s (40.12% Gen, 55.63% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 28m 50s. Estimated total time: 20h 2m 46s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 5s, 500 more iterations: 3h 20m 27s.
[2025-11-13 09:37:44,180][__main__][INFO] - Starting iteration 258.
[2025-11-13 09:37:44,183][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:37:44,183][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:37:54,740][__main__][INFO] - Number of regex retries in iteration 258: 0
[2025-11-13 09:37:54,741][__main__][INFO] - agents played in iteration 258 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:37:55,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,235][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:37:55,318][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:37:55,318][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:37:56,123][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:37:56,420][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:37:56,750][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:37:57,076][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:37:57,403][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:37:57,729][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:37:58,056][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:37:58,383][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:37:58,713][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:37:59,042][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:37:59,369][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:37:59,697][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:38:00,025][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:38:00,354][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:38:00,686][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:38:01,011][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:38:01,339][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:38:01,664][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:38:01,994][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:38:02,316][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:38:02,645][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:38:02,979][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:38:03,305][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:38:03,633][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:38:03,965][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:38:04,294][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:38:04,621][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:38:04,949][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:38:05,275][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:38:05,602][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:38:05,930][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:38:06,267][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:38:06,594][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:38:07,340][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:38:08,106][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:08,108][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:08,110][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:09,222][__main__][INFO] - Iteration 259 took 25s (42.16% Gen, 53.39% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 17m 38s. Estimated total time: 20h 51m 58s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 43s, 500 more iterations: 3h 28m 39s.
[2025-11-13 09:38:09,224][__main__][INFO] - Starting iteration 259.
[2025-11-13 09:38:09,227][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:38:09,227][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:38:19,742][__main__][INFO] - Number of regex retries in iteration 259: 0
[2025-11-13 09:38:19,743][__main__][INFO] - agents played in iteration 259 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:38:20,206][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:20,327][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:38:20,328][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:38:21,127][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:38:21,425][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:38:21,754][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:38:22,084][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:38:22,410][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:38:22,736][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:38:23,063][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:38:23,394][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:38:23,720][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:38:24,047][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:38:24,377][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:38:24,699][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:38:25,026][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:38:25,353][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:38:25,679][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:38:26,006][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:38:26,333][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:38:26,659][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:38:26,990][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:38:27,317][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:38:27,651][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:38:27,980][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:38:28,307][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:38:28,633][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:38:28,964][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:38:29,290][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:38:29,616][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:38:29,943][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:38:30,270][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:38:30,596][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:38:30,929][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:38:31,258][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:38:31,594][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:38:32,367][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:38:33,126][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:33,128][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:33,129][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:34,113][__main__][INFO] - Iteration 260 took 24s (42.25% Gen, 53.79% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 9m 36s. Estimated total time: 20h 44m 21s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 28s, 500 more iterations: 3h 27m 23s.
[2025-11-13 09:38:34,115][__main__][INFO] - Starting iteration 260.
[2025-11-13 09:38:34,118][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 25 and human policies 1.
[2025-11-13 09:38:34,118][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:38:44,109][__main__][INFO] - Number of regex retries in iteration 260: 0
[2025-11-13 09:38:44,110][__main__][INFO] - agents played in iteration 260 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:38:44,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,645][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:38:44,686][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:38:44,686][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:38:45,456][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:38:45,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:38:46,086][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:38:46,413][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:38:46,740][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:38:47,066][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:38:47,393][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:38:47,720][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:38:48,047][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:38:48,373][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:38:48,699][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:38:49,025][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:38:49,352][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:38:49,690][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:38:50,017][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:38:50,350][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:38:50,676][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:38:51,005][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:38:51,332][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:38:51,659][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:38:51,985][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:38:52,311][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:38:52,637][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:38:52,963][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:38:53,293][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:38:53,622][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:38:53,947][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:38:54,274][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:38:54,600][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:38:54,927][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:38:55,254][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:38:55,581][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:38:55,908][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:38:56,651][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:38:57,418][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:38:57,420][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:38:57,421][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:38:59,504][__main__][INFO] - Iteration 261 took 25s (39.35% Gen, 52.43% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 34m 10s. Estimated total time: 21h 9m 21s. Time estimates for 10 more iterations: 4m 13s, 100 more iterations: 42m 18s, 500 more iterations: 3h 31m 33s.
[2025-11-13 09:38:59,506][__main__][INFO] - Starting iteration 261.
[2025-11-13 09:38:59,509][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:38:59,510][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:10,104][__main__][INFO] - Number of regex retries in iteration 261: 0
[2025-11-13 09:39:10,105][__main__][INFO] - agents played in iteration 261 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:39:10,553][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:10,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:10,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:10,673][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:10,673][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:10,674][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:39:11,388][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:39:11,687][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:39:12,013][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:39:12,339][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:39:12,668][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:39:13,002][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:39:13,325][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:39:13,656][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:39:13,982][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:39:14,312][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:39:14,643][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:39:14,976][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:39:15,311][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:39:15,643][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:39:15,972][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:39:16,298][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:39:16,626][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:39:16,956][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:39:17,282][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:39:17,608][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:39:17,934][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:39:18,264][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:39:18,599][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:39:18,928][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:39:19,256][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:39:19,586][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:39:19,915][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:39:20,248][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:39:20,582][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:39:20,914][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:39:21,248][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:39:21,577][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:39:21,911][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:39:22,699][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:39:23,457][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:39:23,459][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:39:23,461][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:39:24,620][__main__][INFO] - Iteration 262 took 25s (42.19% Gen, 53.19% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 19m 59s. Estimated total time: 20h 55m 35s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 51s, 500 more iterations: 3h 29m 15s.
[2025-11-13 09:39:24,622][__main__][INFO] - Starting iteration 262.
[2025-11-13 09:39:24,625][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:39:24,625][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:34,898][__main__][INFO] - Number of regex retries in iteration 262: 0
[2025-11-13 09:39:34,898][__main__][INFO] - agents played in iteration 262 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:39:35,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:35,410][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:35,451][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:35,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:35,492][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:35,492][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:39:36,258][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:39:36,556][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:39:36,896][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:39:37,230][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:39:37,563][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:39:37,893][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:39:38,219][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:39:38,545][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:39:38,879][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:39:39,210][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:39:39,535][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:39:39,864][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:39:40,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:39:40,523][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:39:40,845][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:39:41,175][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:39:41,502][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:39:41,829][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:39:42,160][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:39:42,487][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:39:42,822][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:39:43,156][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:39:43,486][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:39:43,816][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:39:44,149][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:39:44,484][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:39:44,816][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:39:45,146][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:39:45,473][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:39:45,802][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:39:46,129][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:39:46,459][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:39:46,787][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:39:47,565][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:39:48,335][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:39:48,336][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:39:48,338][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:39:49,325][__main__][INFO] - Iteration 263 took 24s (41.59% Gen, 54.41% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 59m 2s. Estimated total time: 20h 35m 3s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 10s, 500 more iterations: 3h 25m 50s.
[2025-11-13 09:39:49,327][__main__][INFO] - Starting iteration 263.
[2025-11-13 09:39:49,330][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:39:49,330][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:39:59,376][__main__][INFO] - Number of regex retries in iteration 263: 0
[2025-11-13 09:39:59,377][__main__][INFO] - agents played in iteration 263 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:39:59,836][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:39:59,957][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:39:59,958][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:00,765][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:40:01,064][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:40:01,390][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:40:01,716][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:40:02,043][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:40:02,370][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:40:02,697][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:40:03,026][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:40:03,354][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:40:03,680][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:40:04,006][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:40:04,333][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:40:04,659][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:40:04,988][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:40:05,316][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:40:05,642][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:40:05,968][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:40:06,296][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:40:06,622][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:40:06,948][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:40:07,280][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:40:07,606][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:40:07,933][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:40:08,260][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:40:08,587][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:40:08,916][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:40:09,244][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:40:09,576][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:40:09,903][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:40:10,231][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:40:10,557][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:40:10,897][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:40:11,226][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:40:12,025][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:40:12,795][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:40:12,796][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:40:12,798][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:40:13,810][__main__][INFO] - Iteration 264 took 24s (41.04% Gen, 54.82% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 47m 37s. Estimated total time: 20h 24m 2s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 48s, 500 more iterations: 3h 24m 0s.
[2025-11-13 09:40:13,812][__main__][INFO] - Starting iteration 264.
[2025-11-13 09:40:13,815][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:40:13,815][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:40:23,937][__main__][INFO] - Number of regex retries in iteration 264: 0
[2025-11-13 09:40:23,938][__main__][INFO] - agents played in iteration 264 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:40:24,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:24,428][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:24,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:24,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:24,510][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:40:24,510][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:25,296][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:40:25,595][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:40:25,922][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:40:26,249][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:40:26,575][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:40:26,901][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:40:27,231][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:40:27,557][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:40:27,885][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:40:28,211][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:40:28,537][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:40:28,867][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:40:29,194][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:40:29,522][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:40:29,849][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:40:30,176][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:40:30,509][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:40:30,832][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:40:31,160][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:40:31,487][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:40:31,817][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:40:32,142][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:40:32,469][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:40:32,795][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:40:33,125][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:40:33,452][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:40:33,779][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:40:34,106][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:40:34,433][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:40:34,766][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:40:35,095][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:40:35,421][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:40:35,749][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:40:36,485][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:40:37,243][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:40:37,244][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:40:37,246][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:40:38,242][__main__][INFO] - Iteration 265 took 24s (41.44% Gen, 54.48% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 44m 35s. Estimated total time: 20h 21m 24s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 42s, 500 more iterations: 3h 23m 34s.
[2025-11-13 09:40:38,244][__main__][INFO] - Starting iteration 265.
[2025-11-13 09:40:38,247][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:40:38,248][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:40:48,858][__main__][INFO] - Number of regex retries in iteration 265: 0
[2025-11-13 09:40:48,859][__main__][INFO] - agents played in iteration 265 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:40:49,323][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:40:49,447][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:40:49,447][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:40:50,249][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:40:50,548][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:40:50,880][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:40:51,208][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:40:51,538][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:40:51,864][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:40:52,191][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:40:52,520][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:40:52,850][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:40:53,184][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:40:53,514][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:40:53,848][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:40:54,178][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:40:54,505][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:40:54,833][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:40:55,167][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:40:55,499][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:40:55,826][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:40:56,153][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:40:56,480][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:40:56,807][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:40:57,134][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:40:57,461][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:40:57,788][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:40:58,116][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:40:58,443][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:40:58,769][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:40:59,097][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:40:59,426][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:40:59,753][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:41:00,088][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:41:00,417][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:41:00,746][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:41:01,526][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:41:02,305][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:02,307][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:02,308][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:03,377][__main__][INFO] - Iteration 266 took 25s (42.22% Gen, 53.52% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 19m 16s. Estimated total time: 20h 56m 31s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 53s, 500 more iterations: 3h 29m 25s.
[2025-11-13 09:41:03,379][__main__][INFO] - Starting iteration 266.
[2025-11-13 09:41:03,382][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:41:03,382][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:41:13,973][__main__][INFO] - Number of regex retries in iteration 266: 0
[2025-11-13 09:41:13,973][__main__][INFO] - agents played in iteration 266 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:41:14,437][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:14,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:14,518][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:14,559][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:14,560][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:41:14,560][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:41:15,331][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:41:15,630][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:41:15,956][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:41:16,281][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:41:16,607][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:41:16,935][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:41:17,261][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:41:17,591][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:41:17,919][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:41:18,247][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:41:18,574][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:41:18,902][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:41:19,228][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:41:19,558][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:41:19,887][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:41:20,214][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:41:20,542][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:41:20,869][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:41:21,196][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:41:21,522][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:41:21,849][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:41:22,179][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:41:22,506][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:41:22,834][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:41:23,160][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:41:23,487][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:41:23,814][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:41:24,143][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:41:24,472][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:41:24,798][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:41:25,126][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:41:25,455][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:41:25,793][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:41:26,570][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:41:27,340][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:27,341][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:27,343][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:28,329][__main__][INFO] - Iteration 267 took 24s (42.45% Gen, 53.59% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 9m 46s. Estimated total time: 20h 47m 25s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 34s, 500 more iterations: 3h 27m 54s.
[2025-11-13 09:41:28,331][__main__][INFO] - Starting iteration 267.
[2025-11-13 09:41:28,335][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:41:28,335][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:41:38,929][__main__][INFO] - Number of regex retries in iteration 267: 0
[2025-11-13 09:41:38,930][__main__][INFO] - agents played in iteration 267 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:41:39,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,489][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:41:39,530][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:41:39,530][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:41:40,316][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:41:40,615][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:41:40,940][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:41:41,266][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:41:41,593][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:41:41,919][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:41:42,247][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:41:42,574][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:41:42,899][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:41:43,225][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:41:43,552][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:41:43,878][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:41:44,205][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:41:44,533][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:41:44,865][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:41:45,193][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:41:45,519][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:41:45,846][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:41:46,173][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:41:46,500][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:41:46,836][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:41:47,163][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:41:47,490][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:41:47,818][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:41:48,148][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:41:48,482][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:41:48,808][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:41:49,139][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:41:49,467][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:41:49,796][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:41:50,123][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:41:50,450][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:41:50,778][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:41:51,520][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:41:52,286][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:41:52,288][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:41:52,289][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:41:53,283][__main__][INFO] - Iteration 268 took 24s (42.46% Gen, 53.55% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 9m 23s. Estimated total time: 20h 47m 27s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 34s, 500 more iterations: 3h 27m 54s.
[2025-11-13 09:41:53,285][__main__][INFO] - Starting iteration 268.
[2025-11-13 09:41:53,288][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:41:53,288][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:42:03,409][__main__][INFO] - Number of regex retries in iteration 268: 0
[2025-11-13 09:42:03,410][__main__][INFO] - agents played in iteration 268 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:42:03,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:03,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:03,959][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:03,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:04,000][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:42:04,000][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:42:04,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:42:05,091][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:42:05,419][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:42:05,746][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:42:06,073][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:42:06,400][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:42:06,725][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:42:07,052][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:42:07,382][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:42:07,711][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:42:08,037][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:42:08,365][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:42:08,693][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:42:09,020][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:42:09,348][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:42:09,675][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:42:10,002][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:42:10,328][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:42:10,655][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:42:10,983][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:42:11,317][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:42:11,643][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:42:11,970][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:42:12,297][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:42:12,629][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:42:12,954][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:42:13,282][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:42:13,609][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:42:13,936][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:42:14,263][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:42:14,591][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:42:14,921][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:42:15,247][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:42:15,985][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:42:16,751][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:42:16,752][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:42:16,754][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:42:17,800][__main__][INFO] - Iteration 269 took 24s (41.29% Gen, 54.44% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 47m 10s. Estimated total time: 20h 25m 39s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 51s, 500 more iterations: 3h 24m 16s.
[2025-11-13 09:42:17,802][__main__][INFO] - Starting iteration 269.
[2025-11-13 09:42:17,805][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:42:17,806][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:42:27,922][__main__][INFO] - Number of regex retries in iteration 269: 0
[2025-11-13 09:42:27,923][__main__][INFO] - agents played in iteration 269 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:42:28,379][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,460][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,500][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:28,501][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:42:28,501][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:42:29,284][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:42:29,583][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:42:29,912][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:42:30,244][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:42:30,571][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:42:30,900][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:42:31,238][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:42:31,565][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:42:31,894][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:42:32,227][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:42:32,561][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:42:32,890][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:42:33,218][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:42:33,552][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:42:33,876][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:42:34,203][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:42:34,538][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:42:34,864][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:42:35,191][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:42:35,518][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:42:35,846][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:42:36,173][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:42:36,502][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:42:36,833][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:42:37,159][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:42:37,486][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:42:37,821][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:42:38,148][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:42:38,475][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:42:38,802][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:42:39,143][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:42:39,470][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:42:39,797][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:42:40,526][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:42:41,274][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:42:41,275][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:42:41,277][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:42:42,414][__main__][INFO] - Iteration 270 took 24s (41.11% Gen, 54.27% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 51m 33s. Estimated total time: 20h 30m 27s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 0s, 500 more iterations: 3h 25m 4s.
[2025-11-13 09:42:42,416][__main__][INFO] - Starting iteration 270.
[2025-11-13 09:42:42,419][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 26 and human policies 1.
[2025-11-13 09:42:42,420][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:42:52,274][__main__][INFO] - Number of regex retries in iteration 270: 0
[2025-11-13 09:42:52,274][__main__][INFO] - agents played in iteration 270 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:42:52,709][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:52,748][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:52,788][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:52,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:42:52,828][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:42:52,829][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:42:53,623][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:42:53,922][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:42:54,251][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:42:54,579][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:42:54,911][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:42:55,237][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:42:55,564][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:42:55,892][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:42:56,218][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:42:56,546][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:42:56,872][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:42:57,201][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:42:57,527][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:42:57,855][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:42:58,181][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:42:58,507][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:42:58,834][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:42:59,162][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:42:59,488][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:42:59,815][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:43:00,143][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:43:00,472][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:43:00,798][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:43:01,125][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:43:01,454][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:43:01,782][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:43:02,108][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:43:02,435][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:43:02,763][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:43:03,090][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:43:03,419][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:43:03,747][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:43:04,073][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:43:04,755][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:43:05,524][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:05,526][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:05,527][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:08,061][__main__][INFO] - Iteration 271 took 25s (38.43% Gen, 51.68% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 42m 50s. Estimated total time: 21h 22m 9s. Time estimates for 10 more iterations: 4m 16s, 100 more iterations: 42m 44s, 500 more iterations: 3h 33m 41s.
[2025-11-13 09:43:08,064][__main__][INFO] - Starting iteration 271.
[2025-11-13 09:43:08,067][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:43:08,068][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:43:18,479][__main__][INFO] - Number of regex retries in iteration 271: 0
[2025-11-13 09:43:18,480][__main__][INFO] - agents played in iteration 271 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:43:18,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:18,990][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:19,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:19,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:19,072][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:43:19,072][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:43:19,825][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:43:20,122][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:43:20,450][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:43:20,776][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:43:21,103][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:43:21,430][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:43:21,764][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:43:22,097][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:43:22,423][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:43:22,750][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:43:23,078][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:43:23,405][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:43:23,732][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:43:24,058][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:43:24,384][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:43:24,710][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:43:25,036][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:43:25,368][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:43:25,690][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:43:26,017][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:43:26,345][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:43:26,674][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:43:26,998][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:43:27,335][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:43:27,662][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:43:27,990][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:43:28,320][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:43:28,646][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:43:28,972][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:43:29,299][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:43:29,625][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:43:29,954][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:43:30,282][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:43:31,069][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:43:31,824][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:31,826][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:31,828][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:32,808][__main__][INFO] - Iteration 272 took 24s (42.08% Gen, 53.95% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 57m 23s. Estimated total time: 20h 37m 7s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 14s, 500 more iterations: 3h 26m 11s.
[2025-11-13 09:43:32,811][__main__][INFO] - Starting iteration 272.
[2025-11-13 09:43:32,813][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:43:32,814][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:43:43,468][__main__][INFO] - Number of regex retries in iteration 272: 0
[2025-11-13 09:43:43,469][__main__][INFO] - agents played in iteration 272 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:43:43,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:43,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:43:44,081][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:43:44,081][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:43:44,869][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:43:45,167][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:43:45,495][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:43:45,824][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:43:46,153][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:43:46,486][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:43:46,813][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:43:47,144][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:43:47,472][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:43:47,802][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:43:48,142][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:43:48,472][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:43:48,798][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:43:49,125][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:43:49,451][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:43:49,779][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:43:50,105][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:43:50,434][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:43:50,766][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:43:51,099][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:43:51,428][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:43:51,762][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:43:52,083][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:43:52,410][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:43:52,737][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:43:53,067][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:43:53,389][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:43:53,718][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:43:54,043][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:43:54,372][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:43:54,696][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:43:55,023][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:43:55,350][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:43:56,135][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:43:56,895][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:43:56,896][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:43:56,898][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:43:57,910][__main__][INFO] - Iteration 273 took 25s (42.45% Gen, 53.51% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 14m 43s. Estimated total time: 20h 54m 52s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 49s, 500 more iterations: 3h 29m 8s.
[2025-11-13 09:43:57,912][__main__][INFO] - Starting iteration 273.
[2025-11-13 09:43:57,915][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:43:57,916][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:07,831][__main__][INFO] - Number of regex retries in iteration 273: 0
[2025-11-13 09:44:07,831][__main__][INFO] - agents played in iteration 273 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:44:08,274][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:08,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:08,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:08,394][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:08,395][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:08,395][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:44:09,200][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:44:09,500][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:44:09,828][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:44:10,155][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:44:10,488][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:44:10,817][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:44:11,145][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:44:11,471][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:44:11,799][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:44:12,133][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:44:12,464][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:44:12,792][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:44:13,118][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:44:13,443][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:44:13,771][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:44:14,099][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:44:14,445][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:44:14,780][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:44:15,113][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:44:15,446][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:44:15,781][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:44:16,110][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:44:16,436][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:44:16,766][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:44:17,100][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:44:17,427][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:44:17,754][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:44:18,083][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:44:18,423][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:44:18,753][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:44:19,079][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:44:19,406][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:44:19,741][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:44:20,448][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:44:21,222][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:44:21,223][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:44:21,225][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:44:22,235][__main__][INFO] - Iteration 274 took 24s (40.77% Gen, 55.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 35m 29s. Estimated total time: 20h 16m 3s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 32s, 500 more iterations: 3h 22m 40s.
[2025-11-13 09:44:22,238][__main__][INFO] - Starting iteration 274.
[2025-11-13 09:44:22,241][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:44:22,241][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:33,028][__main__][INFO] - Number of regex retries in iteration 274: 0
[2025-11-13 09:44:33,029][__main__][INFO] - agents played in iteration 274 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:44:33,488][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:33,528][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:33,568][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:33,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:33,608][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:33,609][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:44:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:44:34,698][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:44:35,026][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:44:35,353][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:44:35,680][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:44:36,008][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:44:36,336][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:44:36,662][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:44:36,991][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:44:37,317][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:44:37,643][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:44:37,969][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:44:38,296][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:44:38,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:44:38,953][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:44:39,280][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:44:39,607][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:44:39,936][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:44:40,264][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:44:40,591][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:44:40,917][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:44:41,245][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:44:41,573][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:44:41,901][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:44:42,232][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:44:42,558][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:44:42,889][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:44:43,215][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:44:43,544][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:44:43,870][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:44:44,198][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:44:44,524][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:44:44,853][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:44:45,556][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:44:46,326][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:44:46,328][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:44:46,330][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:44:47,445][__main__][INFO] - Iteration 275 took 25s (42.80% Gen, 52.77% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 19m 17s. Estimated total time: 21h 0m 16s. Time estimates for 10 more iterations: 4m 12s, 100 more iterations: 42m 0s, 500 more iterations: 3h 30m 2s.
[2025-11-13 09:44:47,448][__main__][INFO] - Starting iteration 275.
[2025-11-13 09:44:47,451][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:44:47,451][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:44:58,104][__main__][INFO] - Number of regex retries in iteration 275: 0
[2025-11-13 09:44:58,105][__main__][INFO] - agents played in iteration 275 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:44:58,564][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:58,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:58,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:58,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:44:58,684][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:44:58,685][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:44:59,476][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:44:59,774][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:45:00,107][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:45:00,436][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:45:00,764][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:45:01,094][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:45:01,420][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:45:01,752][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:45:02,078][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:45:02,412][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:45:02,742][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:45:03,077][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:45:03,410][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:45:03,736][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:45:04,063][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:45:04,390][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:45:04,716][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:45:05,042][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:45:05,369][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:45:05,697][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:45:06,024][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:45:06,351][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:45:06,678][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:45:07,005][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:45:07,332][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:45:07,660][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:45:07,986][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:45:08,313][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:45:08,640][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:45:08,967][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:45:09,295][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:45:09,622][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:45:09,964][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:45:10,718][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:45:11,495][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:45:11,497][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:45:11,498][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:45:12,527][__main__][INFO] - Iteration 276 took 25s (42.48% Gen, 53.41% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 12m 27s. Estimated total time: 20h 53m 50s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 47s, 500 more iterations: 3h 28m 58s.
[2025-11-13 09:45:12,529][__main__][INFO] - Starting iteration 276.
[2025-11-13 09:45:12,532][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:45:12,533][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:45:23,360][__main__][INFO] - Number of regex retries in iteration 276: 0
[2025-11-13 09:45:23,360][__main__][INFO] - agents played in iteration 276 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:45:23,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,864][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,904][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:23,946][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:45:23,946][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:45:24,747][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:45:25,044][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:45:25,375][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:45:25,707][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:45:26,036][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:45:26,363][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:45:26,693][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:45:27,019][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:45:27,346][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:45:27,673][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:45:28,005][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:45:28,338][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:45:28,664][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:45:28,992][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:45:29,326][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:45:29,655][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:45:29,985][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:45:30,312][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:45:30,639][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:45:30,972][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:45:31,299][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:45:31,625][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:45:31,952][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:45:32,279][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:45:32,608][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:45:32,935][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:45:33,266][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:45:33,594][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:45:33,920][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:45:34,247][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:45:34,576][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:45:34,903][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:45:35,231][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:45:35,998][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:45:36,759][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:45:36,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:45:36,762][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:45:37,726][__main__][INFO] - Iteration 277 took 25s (42.97% Gen, 53.19% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 17m 55s. Estimated total time: 20h 59m 44s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 59s, 500 more iterations: 3h 29m 57s.
[2025-11-13 09:45:37,728][__main__][INFO] - Starting iteration 277.
[2025-11-13 09:45:37,731][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:45:37,732][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:45:48,245][__main__][INFO] - Number of regex retries in iteration 277: 0
[2025-11-13 09:45:48,246][__main__][INFO] - agents played in iteration 277 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:45:48,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:48,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:48,812][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:48,853][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:45:48,853][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:45:48,854][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:45:49,638][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:45:49,935][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:45:50,273][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:45:50,605][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:45:50,932][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:45:51,259][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:45:51,587][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:45:51,914][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:45:52,241][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:45:52,567][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:45:52,894][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:45:53,221][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:45:53,548][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:45:53,875][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:45:54,204][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:45:54,533][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:45:54,862][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:45:55,189][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:45:55,518][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:45:55,844][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:45:56,170][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:45:56,496][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:45:56,823][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:45:57,150][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:45:57,477][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:45:57,804][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:45:58,129][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:45:58,457][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:45:58,783][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:45:59,110][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:45:59,436][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:45:59,766][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:46:00,094][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:46:00,869][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:46:01,619][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:46:01,621][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:46:01,623][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:46:02,597][__main__][INFO] - Iteration 278 took 24s (42.28% Gen, 53.79% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 1m 6s. Estimated total time: 20h 43m 19s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 26s, 500 more iterations: 3h 27m 13s.
[2025-11-13 09:46:02,599][__main__][INFO] - Starting iteration 278.
[2025-11-13 09:46:02,602][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:46:02,602][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:46:13,463][__main__][INFO] - Number of regex retries in iteration 278: 0
[2025-11-13 09:46:13,464][__main__][INFO] - agents played in iteration 278 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:46:13,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,091][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:14,092][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:46:14,092][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:46:14,873][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:46:15,183][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:46:15,509][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:46:15,839][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:46:16,164][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:46:16,495][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:46:16,826][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:46:17,155][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:46:17,482][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:46:17,816][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:46:18,142][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:46:18,475][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:46:18,807][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:46:19,133][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:46:19,463][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:46:19,792][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:46:20,122][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:46:20,449][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:46:20,778][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:46:21,111][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:46:21,442][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:46:21,777][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:46:22,102][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:46:22,429][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:46:22,755][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:46:23,083][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:46:23,410][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:46:23,740][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:46:24,069][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:46:24,401][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:46:24,729][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:46:25,063][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:46:25,398][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:46:26,166][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:46:26,930][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:46:26,932][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:46:26,933][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:46:28,003][__main__][INFO] - Iteration 279 took 25s (42.76% Gen, 53.03% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 27m 28s. Estimated total time: 21h 10m 7s. Time estimates for 10 more iterations: 4m 14s, 100 more iterations: 42m 20s, 500 more iterations: 3h 31m 41s.
[2025-11-13 09:46:28,005][__main__][INFO] - Starting iteration 279.
[2025-11-13 09:46:28,009][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:46:28,009][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:46:38,489][__main__][INFO] - Number of regex retries in iteration 279: 0
[2025-11-13 09:46:38,490][__main__][INFO] - agents played in iteration 279 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:46:38,985][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,065][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,105][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:46:39,106][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:46:39,106][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:46:39,910][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:46:40,211][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:46:40,538][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:46:40,867][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:46:41,201][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:46:41,527][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:46:41,854][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:46:42,180][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:46:42,507][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:46:42,833][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:46:43,160][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:46:43,485][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:46:43,816][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:46:44,141][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:46:44,468][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:46:44,795][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:46:45,123][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:46:45,447][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:46:45,774][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:46:46,100][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:46:46,427][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:46:46,756][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:46:47,083][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:46:47,410][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:46:47,736][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:46:48,062][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:46:48,388][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:46:48,716][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:46:49,042][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:46:49,368][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:46:49,698][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:46:50,024][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:46:50,352][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:46:51,137][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:46:51,915][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:46:51,917][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:46:51,918][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:46:53,018][__main__][INFO] - Iteration 280 took 25s (41.91% Gen, 53.68% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 7m 26s. Estimated total time: 20h 50m 30s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 41s, 500 more iterations: 3h 28m 25s.
[2025-11-13 09:46:53,020][__main__][INFO] - Starting iteration 280.
[2025-11-13 09:46:53,023][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 27 and human policies 1.
[2025-11-13 09:46:53,023][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:03,776][__main__][INFO] - Number of regex retries in iteration 280: 0
[2025-11-13 09:47:03,776][__main__][INFO] - agents played in iteration 280 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:47:04,239][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:04,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:04,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:04,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:04,360][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:04,361][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:05,133][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:47:05,432][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:47:05,763][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:47:06,096][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:47:06,425][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:47:06,751][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:47:07,077][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:47:07,406][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:47:07,734][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:47:08,067][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:47:08,396][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:47:08,727][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:47:09,049][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:47:09,374][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:47:09,701][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:47:10,028][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:47:10,354][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:47:10,680][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:47:11,007][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:47:11,334][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:47:11,662][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:47:11,988][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:47:12,316][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:47:12,642][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:47:12,969][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:47:13,296][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:47:13,622][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:47:13,949][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:47:14,283][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:47:14,610][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:47:14,940][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:47:15,273][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:47:15,614][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:47:16,371][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:47:17,103][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:47:17,105][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:47:17,106][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:47:19,111][__main__][INFO] - Iteration 281 took 26s (41.22% Gen, 51.10% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 0m 58s. Estimated total time: 21h 44m 29s. Time estimates for 10 more iterations: 4m 20s, 100 more iterations: 43m 28s, 500 more iterations: 3h 37m 24s.
[2025-11-13 09:47:19,114][__main__][INFO] - Starting iteration 281.
[2025-11-13 09:47:19,116][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:47:19,117][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:30,004][__main__][INFO] - Number of regex retries in iteration 281: 0
[2025-11-13 09:47:30,005][__main__][INFO] - agents played in iteration 281 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:47:30,443][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:30,485][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:30,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:30,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:30,564][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:30,564][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:31,282][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:47:31,578][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:47:31,905][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:47:32,231][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:47:32,558][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:47:32,886][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:47:33,212][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:47:33,541][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:47:33,868][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:47:34,194][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:47:34,520][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:47:34,849][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:47:35,176][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:47:35,501][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:47:35,829][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:47:36,156][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:47:36,483][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:47:36,816][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:47:37,143][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:47:37,470][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:47:37,796][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:47:38,124][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:47:38,451][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:47:38,779][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:47:39,107][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:47:39,440][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:47:39,771][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:47:40,104][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:47:40,435][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:47:40,763][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:47:41,091][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:47:41,418][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:47:41,746][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:47:42,507][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:47:43,250][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:47:43,252][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:47:43,254][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:47:44,149][__main__][INFO] - Iteration 282 took 25s (43.49% Gen, 52.92% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 7m 44s. Estimated total time: 20h 51m 40s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 43s, 500 more iterations: 3h 28m 36s.
[2025-11-13 09:47:44,151][__main__][INFO] - Starting iteration 282.
[2025-11-13 09:47:44,154][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:47:44,155][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:47:54,468][__main__][INFO] - Number of regex retries in iteration 282: 0
[2025-11-13 09:47:54,469][__main__][INFO] - agents played in iteration 282 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:47:54,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:54,963][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:55,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:55,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:47:55,044][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:47:55,044][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:47:55,781][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:47:56,077][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:47:56,403][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:47:56,736][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:47:57,061][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:47:57,388][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:47:57,715][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:47:58,044][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:47:58,370][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:47:58,696][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:47:59,025][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:47:59,351][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:47:59,677][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:48:00,003][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:48:00,334][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:48:00,656][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:48:00,983][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:48:01,309][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:48:01,640][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:48:01,961][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:48:02,290][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:48:02,617][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:48:02,948][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:48:03,272][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:48:03,599][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:48:03,930][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:48:04,259][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:48:04,586][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:48:04,914][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:48:05,242][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:48:05,572][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:48:05,910][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:48:06,238][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:48:07,000][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:48:07,731][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:07,732][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:07,734][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:08,633][__main__][INFO] - Iteration 283 took 24s (42.13% Gen, 54.19% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 39m 40s. Estimated total time: 20h 24m 0s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 48s, 500 more iterations: 3h 24m 0s.
[2025-11-13 09:48:08,635][__main__][INFO] - Starting iteration 283.
[2025-11-13 09:48:08,638][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:48:08,639][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:48:18,547][__main__][INFO] - Number of regex retries in iteration 283: 0
[2025-11-13 09:48:18,548][__main__][INFO] - agents played in iteration 283 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:48:19,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:19,143][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:48:19,143][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:48:19,914][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:48:20,212][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:48:20,538][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:48:20,864][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:48:21,192][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:48:21,516][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:48:21,849][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:48:22,176][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:48:22,504][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:48:22,831][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:48:23,164][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:48:23,494][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:48:23,820][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:48:24,158][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:48:24,484][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:48:24,813][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:48:25,143][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:48:25,470][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:48:25,797][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:48:26,130][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:48:26,460][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:48:26,792][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:48:27,129][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:48:27,463][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:48:27,795][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:48:28,124][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:48:28,453][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:48:28,781][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:48:29,108][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:48:29,437][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:48:29,764][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:48:30,091][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:48:30,418][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:48:31,201][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:48:31,918][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:31,919][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:31,921][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:32,876][__main__][INFO] - Iteration 284 took 24s (40.88% Gen, 55.17% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 27m 11s. Estimated total time: 20h 11m 55s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 23s, 500 more iterations: 3h 21m 59s.
[2025-11-13 09:48:32,878][__main__][INFO] - Starting iteration 284.
[2025-11-13 09:48:32,881][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:48:32,881][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:48:43,241][__main__][INFO] - Number of regex retries in iteration 284: 0
[2025-11-13 09:48:43,242][__main__][INFO] - agents played in iteration 284 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:48:43,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:43,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:43,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:43,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:48:43,869][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:48:43,869][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:48:44,654][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:48:44,951][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:48:45,281][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:48:45,608][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:48:45,935][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:48:46,262][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:48:46,588][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:48:46,924][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:48:47,250][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:48:47,576][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:48:47,903][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:48:48,245][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:48:48,573][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:48:48,900][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:48:49,233][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:48:49,558][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:48:49,889][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:48:50,218][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:48:50,543][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:48:50,872][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:48:51,198][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:48:51,525][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:48:51,851][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:48:52,178][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:48:52,504][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:48:52,832][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:48:53,157][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:48:53,485][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:48:53,812][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:48:54,137][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:48:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:48:54,792][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:48:55,119][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:48:55,871][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:48:56,599][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:48:56,601][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:48:56,602][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:48:57,861][__main__][INFO] - Iteration 285 took 24s (41.47% Gen, 53.48% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 3m 56s. Estimated total time: 20h 49m 5s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 38s, 500 more iterations: 3h 28m 10s.
[2025-11-13 09:48:57,863][__main__][INFO] - Starting iteration 285.
[2025-11-13 09:48:57,866][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:48:57,867][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:08,227][__main__][INFO] - Number of regex retries in iteration 285: 0
[2025-11-13 09:49:08,228][__main__][INFO] - agents played in iteration 285 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:49:08,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:08,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:08,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:08,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:08,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:08,806][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:49:09,581][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:49:09,880][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:49:10,213][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:49:10,544][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:49:10,873][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:49:11,199][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:49:11,526][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:49:11,857][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:49:12,186][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:49:12,515][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:49:12,842][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:49:13,168][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:49:13,496][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:49:13,822][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:49:14,152][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:49:14,478][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:49:14,805][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:49:15,139][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:49:15,465][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:49:15,793][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:49:16,120][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:49:16,447][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:49:16,774][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:49:17,101][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:49:17,428][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:49:17,755][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:49:18,084][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:49:18,425][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:49:18,753][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:49:19,082][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:49:19,415][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:49:19,756][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:49:20,085][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:49:20,853][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:49:21,563][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:49:21,564][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:49:21,566][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:49:22,503][__main__][INFO] - Iteration 286 took 24s (42.05% Gen, 54.14% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 46m 21s. Estimated total time: 20h 31m 55s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 3s, 500 more iterations: 3h 25m 19s.
[2025-11-13 09:49:22,505][__main__][INFO] - Starting iteration 286.
[2025-11-13 09:49:22,508][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:49:22,509][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:32,615][__main__][INFO] - Number of regex retries in iteration 286: 0
[2025-11-13 09:49:32,616][__main__][INFO] - agents played in iteration 286 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:49:33,053][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:33,176][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:33,176][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:49:33,951][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:49:34,252][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:49:34,579][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:49:34,907][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:49:35,233][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:49:35,561][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:49:35,888][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:49:36,215][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:49:36,544][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:49:36,872][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:49:37,201][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:49:37,535][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:49:37,864][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:49:38,193][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:49:38,520][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:49:38,849][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:49:39,182][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:49:39,512][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:49:39,846][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:49:40,176][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:49:40,503][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:49:40,846][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:49:41,176][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:49:41,506][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:49:41,840][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:49:42,175][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:49:42,505][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:49:42,835][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:49:43,169][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:49:43,495][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:49:43,821][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:49:44,155][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:49:44,483][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:49:45,224][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:49:45,972][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:49:46,112][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:49:46,114][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:49:47,180][__main__][INFO] - Iteration 287 took 24s (40.96% Gen, 54.71% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 47m 38s. Estimated total time: 20h 33m 37s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 7s, 500 more iterations: 3h 25m 36s.
[2025-11-13 09:49:47,182][__main__][INFO] - Starting iteration 287.
[2025-11-13 09:49:47,185][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:49:47,186][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:49:58,348][__main__][INFO] - Number of regex retries in iteration 287: 0
[2025-11-13 09:49:58,348][__main__][INFO] - agents played in iteration 287 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:49:58,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:58,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:58,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:58,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:49:58,920][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:49:58,920][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:49:59,687][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:49:59,986][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:50:00,313][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:50:00,639][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:50:00,971][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:50:01,296][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:50:01,624][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:50:01,950][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:50:02,277][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:50:02,603][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:50:02,930][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:50:03,257][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:50:03,584][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:50:03,912][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:50:04,239][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:50:04,566][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:50:04,893][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:50:05,219][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:50:05,548][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:50:05,880][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:50:06,210][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:50:06,536][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:50:06,864][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:50:07,191][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:50:07,517][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:50:07,844][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:50:08,172][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:50:08,502][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:50:08,833][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:50:09,159][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:50:09,487][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:50:09,814][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:50:10,144][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:50:10,906][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:50:11,655][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:50:11,656][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:50:11,658][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:50:12,885][__main__][INFO] - Iteration 288 took 25s (43.43% Gen, 51.79% Train). Generation: 11s, Training: 13s. Estimated remaining time: 19h 38m 38s. Estimated total time: 21h 25m 2s. Time estimates for 10 more iterations: 4m 17s, 100 more iterations: 42m 50s, 500 more iterations: 3h 34m 10s.
[2025-11-13 09:50:12,887][__main__][INFO] - Starting iteration 288.
[2025-11-13 09:50:12,890][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:50:12,891][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:50:23,340][__main__][INFO] - Number of regex retries in iteration 288: 0
[2025-11-13 09:50:23,341][__main__][INFO] - agents played in iteration 288 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:50:23,786][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,827][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:23,910][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:50:23,910][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:50:24,697][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:50:24,995][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:50:25,322][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:50:25,652][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:50:25,980][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:50:26,307][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:50:26,636][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:50:26,966][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:50:27,294][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:50:27,628][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:50:27,955][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:50:28,283][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:50:28,613][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:50:28,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:50:29,281][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:50:29,607][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:50:29,935][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:50:30,263][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:50:30,593][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:50:30,920][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:50:31,249][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:50:31,579][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:50:31,907][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:50:32,247][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:50:32,577][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:50:32,907][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:50:33,234][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:50:33,575][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:50:33,904][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:50:34,235][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:50:34,573][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:50:34,904][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:50:35,234][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:50:36,016][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:50:36,734][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:50:36,735][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:50:36,737][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:50:37,667][__main__][INFO] - Iteration 289 took 24s (42.17% Gen, 54.07% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 52m 6s. Estimated total time: 20h 38m 55s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 17s, 500 more iterations: 3h 26m 29s.
[2025-11-13 09:50:37,670][__main__][INFO] - Starting iteration 289.
[2025-11-13 09:50:37,673][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:50:37,674][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:50:48,406][__main__][INFO] - Number of regex retries in iteration 289: 0
[2025-11-13 09:50:48,407][__main__][INFO] - agents played in iteration 289 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:50:48,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:48,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:48,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:49,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:50:49,016][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:50:49,016][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:50:49,791][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:50:50,088][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:50:50,417][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:50:50,742][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:50:51,070][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:50:51,405][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:50:51,731][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:50:52,065][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:50:52,395][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:50:52,728][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:50:53,055][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:50:53,381][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:50:53,710][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:50:54,038][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:50:54,379][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:50:54,712][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:50:55,047][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:50:55,380][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:50:55,707][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:50:56,033][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:50:56,360][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:50:56,689][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:50:57,022][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:50:57,351][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:50:57,678][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:50:58,008][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:50:58,342][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:50:58,671][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:50:59,000][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:50:59,329][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:50:59,669][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:50:59,996][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:51:00,325][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:51:01,089][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:51:01,810][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:01,811][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:01,813][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:02,741][__main__][INFO] - Iteration 290 took 25s (42.81% Gen, 53.48% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 6m 13s. Estimated total time: 20h 53m 26s. Time estimates for 10 more iterations: 4m 10s, 100 more iterations: 41m 46s, 500 more iterations: 3h 28m 54s.
[2025-11-13 09:51:02,743][__main__][INFO] - Starting iteration 290.
[2025-11-13 09:51:02,747][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 28 and human policies 1.
[2025-11-13 09:51:02,747][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:51:13,351][__main__][INFO] - Number of regex retries in iteration 290: 0
[2025-11-13 09:51:13,352][__main__][INFO] - agents played in iteration 290 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:51:13,809][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,891][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,931][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:13,932][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:51:13,932][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:51:14,722][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:51:15,022][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:51:15,350][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:51:15,677][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:51:16,004][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:51:16,330][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:51:16,657][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:51:16,984][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:51:17,313][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:51:17,642][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:51:17,975][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:51:18,302][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:51:18,629][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:51:18,958][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:51:19,288][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:51:19,617][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:51:19,950][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:51:20,277][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:51:20,603][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:51:20,931][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:51:21,257][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:51:21,586][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:51:21,913][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:51:22,240][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:51:22,566][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:51:22,901][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:51:23,221][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:51:23,549][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:51:23,877][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:51:24,210][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:51:24,534][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:51:24,862][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:51:25,190][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:51:25,977][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:51:26,698][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:26,700][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:26,702][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:28,574][__main__][INFO] - Iteration 291 took 25s (41.06% Gen, 51.69% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 43m 44s. Estimated total time: 21h 31m 24s. Time estimates for 10 more iterations: 4m 18s, 100 more iterations: 43m 2s, 500 more iterations: 3h 35m 14s.
[2025-11-13 09:51:28,576][__main__][INFO] - Starting iteration 291.
[2025-11-13 09:51:28,579][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:51:28,579][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:51:37,932][__main__][INFO] - Number of regex retries in iteration 291: 0
[2025-11-13 09:51:37,933][__main__][INFO] - agents played in iteration 291 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:51:38,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,516][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:51:38,516][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:51:38,517][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:51:39,318][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:51:39,616][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:51:39,943][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:51:40,284][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:51:40,614][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:51:40,947][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:51:41,277][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:51:41,611][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:51:41,938][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:51:42,267][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:51:42,603][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:51:42,928][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:51:43,256][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:51:43,582][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:51:43,914][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:51:44,244][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:51:44,575][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:51:44,902][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:51:45,230][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:51:45,560][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:51:45,888][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:51:46,216][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:51:46,543][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:51:46,875][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:51:47,203][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:51:47,531][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:51:47,862][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:51:48,193][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:51:48,521][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:51:48,849][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:51:49,175][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:51:49,508][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:51:49,836][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:51:50,621][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:51:51,341][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:51:51,343][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:51:51,344][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:51:52,307][__main__][INFO] - Iteration 292 took 23s (39.42% Gen, 56.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 58m 25s. Estimated total time: 19h 46m 28s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 32s, 500 more iterations: 3h 17m 44s.
[2025-11-13 09:51:52,364][__main__][INFO] - Starting iteration 292.
[2025-11-13 09:51:52,368][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:51:52,368][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:01,465][__main__][INFO] - Number of regex retries in iteration 292: 0
[2025-11-13 09:52:01,466][__main__][INFO] - agents played in iteration 292 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:52:01,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:01,957][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:01,996][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:02,036][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:02,036][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:02,037][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:02,798][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:52:03,095][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:52:03,423][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:52:03,750][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:52:04,086][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:52:04,414][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:52:04,741][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:52:05,068][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:52:05,406][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:52:05,734][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:52:06,061][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:52:06,387][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:52:06,716][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:52:07,043][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:52:07,371][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:52:07,702][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:52:08,025][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:52:08,358][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:52:08,685][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:52:09,018][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:52:09,345][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:52:09,673][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:52:10,000][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:52:10,326][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:52:10,653][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:52:10,982][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:52:11,309][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:52:11,639][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:52:11,967][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:52:12,298][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:52:12,624][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:52:12,952][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:52:13,290][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:52:14,012][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:52:14,725][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:52:14,727][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:52:14,729][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:52:15,745][__main__][INFO] - Iteration 293 took 23s (38.91% Gen, 56.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 40m 29s. Estimated total time: 19h 28m 56s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 57s, 500 more iterations: 3h 14m 49s.
[2025-11-13 09:52:15,747][__main__][INFO] - Starting iteration 293.
[2025-11-13 09:52:15,751][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:52:15,752][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:25,999][__main__][INFO] - Number of regex retries in iteration 293: 0
[2025-11-13 09:52:25,999][__main__][INFO] - agents played in iteration 293 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:52:26,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:26,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:26,535][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:26,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:26,575][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:26,576][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:27,309][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:52:27,613][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:52:27,934][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:52:28,262][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:52:28,590][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:52:28,920][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:52:29,247][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:52:29,576][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:52:29,903][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:52:30,232][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:52:30,558][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:52:30,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:52:31,212][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:52:31,538][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:52:31,865][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:52:32,194][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:52:32,522][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:52:32,850][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:52:33,191][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:52:33,518][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:52:33,846][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:52:34,173][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:52:34,510][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:52:34,845][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:52:35,178][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:52:35,505][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:52:35,832][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:52:36,159][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:52:36,486][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:52:36,818][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:52:37,145][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:52:37,471][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:52:37,800][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:52:38,520][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:52:39,230][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:52:39,232][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:52:39,234][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:52:40,171][__main__][INFO] - Iteration 294 took 24s (41.96% Gen, 54.20% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 32m 11s. Estimated total time: 20h 21m 2s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 42s, 500 more iterations: 3h 23m 30s.
[2025-11-13 09:52:40,173][__main__][INFO] - Starting iteration 294.
[2025-11-13 09:52:40,176][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:52:40,176][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:52:49,423][__main__][INFO] - Number of regex retries in iteration 294: 0
[2025-11-13 09:52:49,423][__main__][INFO] - agents played in iteration 294 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:52:49,873][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:49,927][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:49,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:50,021][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:52:50,022][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:52:50,022][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:52:50,771][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:52:51,067][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:52:51,397][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:52:51,727][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:52:52,057][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:52:52,385][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:52:52,714][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:52:53,043][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:52:53,370][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:52:53,697][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:52:54,024][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:52:54,351][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:52:54,679][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:52:55,006][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:52:55,333][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:52:55,661][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:52:55,987][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:52:56,316][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:52:56,648][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:52:56,983][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:52:57,316][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:52:57,650][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:52:57,978][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:52:58,308][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:52:58,635][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:52:58,962][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:52:59,298][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:52:59,625][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:52:59,954][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:53:00,286][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:53:00,620][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:53:00,949][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:53:01,277][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:53:02,024][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:53:02,728][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:02,730][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:02,732][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:03,686][__main__][INFO] - Iteration 295 took 23s (39.33% Gen, 56.61% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 46m 20s. Estimated total time: 19h 35m 35s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 55s.
[2025-11-13 09:53:03,688][__main__][INFO] - Starting iteration 295.
[2025-11-13 09:53:03,692][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:53:03,692][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:53:12,764][__main__][INFO] - Number of regex retries in iteration 295: 0
[2025-11-13 09:53:12,764][__main__][INFO] - agents played in iteration 295 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:53:13,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:13,261][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:13,301][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:13,340][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:13,341][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:53:13,341][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:53:14,091][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:53:14,389][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:53:14,717][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:53:15,045][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:53:15,372][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:53:15,700][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:53:16,026][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:53:16,354][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:53:16,684][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:53:17,013][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:53:17,341][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:53:17,669][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:53:17,996][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:53:18,325][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:53:18,652][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:53:18,991][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:53:19,322][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:53:19,649][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:53:19,978][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:53:20,320][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:53:20,655][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:53:20,988][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:53:21,318][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:53:21,652][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:53:21,986][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:53:22,317][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:53:22,645][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:53:22,972][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:53:23,301][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:53:23,629][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:53:23,957][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:53:24,285][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:53:24,615][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:53:25,369][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:53:26,105][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:26,107][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:26,108][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:27,072][__main__][INFO] - Iteration 296 took 23s (38.80% Gen, 57.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 39m 24s. Estimated total time: 19h 29m 2s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 50s.
[2025-11-13 09:53:27,074][__main__][INFO] - Starting iteration 296.
[2025-11-13 09:53:27,077][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:53:27,078][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:53:36,172][__main__][INFO] - Number of regex retries in iteration 296: 0
[2025-11-13 09:53:36,172][__main__][INFO] - agents played in iteration 296 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:53:36,628][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:36,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:36,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:36,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:53:36,748][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:53:36,748][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:53:37,508][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:53:37,806][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:53:38,134][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:53:38,460][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:53:38,787][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:53:39,116][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:53:39,444][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:53:39,778][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:53:40,112][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:53:40,442][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:53:40,777][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:53:41,107][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:53:41,436][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:53:41,765][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:53:42,095][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:53:42,424][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:53:42,754][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:53:43,081][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:53:43,411][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:53:43,740][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:53:44,076][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:53:44,407][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:53:44,734][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:53:45,061][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:53:45,404][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:53:45,733][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:53:46,063][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:53:46,392][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:53:46,731][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:53:47,057][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:53:47,385][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:53:47,713][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:53:48,041][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:53:48,784][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:53:49,508][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:53:49,510][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:53:49,511][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:53:50,446][__main__][INFO] - Iteration 297 took 23s (38.92% Gen, 57.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 38m 28s. Estimated total time: 19h 28m 29s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 44s.
[2025-11-13 09:53:50,448][__main__][INFO] - Starting iteration 297.
[2025-11-13 09:53:50,451][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:53:50,453][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:53:59,810][__main__][INFO] - Number of regex retries in iteration 297: 0
[2025-11-13 09:53:59,810][__main__][INFO] - agents played in iteration 297 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:54:00,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:00,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:00,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:00,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:00,387][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:00,387][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:01,160][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:54:01,458][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:54:01,785][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:54:02,121][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:54:02,448][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:54:02,780][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:54:03,107][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:54:03,440][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:54:03,761][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:54:04,096][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:54:04,427][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:54:04,756][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:54:05,091][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:54:05,422][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:54:05,750][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:54:06,077][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:54:06,404][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:54:06,732][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:54:07,060][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:54:07,389][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:54:07,730][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:54:08,058][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:54:08,388][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:54:08,717][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:54:09,048][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:54:09,376][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:54:09,704][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:54:10,038][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:54:10,362][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:54:10,691][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:54:11,019][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:54:11,350][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:54:11,676][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:54:12,422][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:54:13,139][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:54:13,140][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:54:13,142][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:54:14,269][__main__][INFO] - Iteration 298 took 23s (39.28% Gen, 55.97% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 0m 31s. Estimated total time: 19h 50m 56s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 29s.
[2025-11-13 09:54:14,271][__main__][INFO] - Starting iteration 298.
[2025-11-13 09:54:14,274][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:54:14,274][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:54:23,230][__main__][INFO] - Number of regex retries in iteration 298: 0
[2025-11-13 09:54:23,231][__main__][INFO] - agents played in iteration 298 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:54:23,690][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:23,730][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:23,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:23,810][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:23,811][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:23,811][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:24,572][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:54:24,871][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:54:25,199][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:54:25,525][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:54:25,852][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:54:26,179][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:54:26,508][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:54:26,836][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:54:27,164][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:54:27,491][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:54:27,818][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:54:28,148][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:54:28,477][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:54:28,804][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:54:29,136][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:54:29,469][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:54:29,798][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:54:30,129][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:54:30,456][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:54:30,785][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:54:31,116][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:54:31,444][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:54:31,774][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:54:32,109][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:54:32,444][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:54:32,772][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:54:33,101][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:54:33,429][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:54:33,759][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:54:34,086][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:54:34,414][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:54:34,743][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:54:35,070][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:54:35,798][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:54:36,526][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:54:36,528][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:54:36,530][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:54:37,493][__main__][INFO] - Iteration 299 took 23s (38.57% Gen, 57.27% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 30m 13s. Estimated total time: 19h 21m 1s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 42s, 500 more iterations: 3h 13m 30s.
[2025-11-13 09:54:37,496][__main__][INFO] - Starting iteration 299.
[2025-11-13 09:54:37,499][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:54:37,499][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:54:46,665][__main__][INFO] - Number of regex retries in iteration 299: 0
[2025-11-13 09:54:46,666][__main__][INFO] - agents played in iteration 299 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:54:47,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:47,171][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:47,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:47,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:54:47,253][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:54:47,253][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:54:48,024][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:54:48,324][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:54:48,652][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:54:48,979][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:54:49,307][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:54:49,640][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:54:49,963][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:54:50,292][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:54:50,618][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:54:50,952][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:54:51,283][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:54:51,612][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:54:51,940][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:54:52,269][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:54:52,599][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:54:52,928][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:54:53,255][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:54:53,583][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:54:53,911][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:54:54,238][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:54:54,568][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:54:54,897][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:54:55,224][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:54:55,557][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:54:55,886][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:54:56,213][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:54:56,540][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:54:56,868][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:54:57,196][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:54:57,523][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:54:57,854][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:54:58,180][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:54:58,508][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:54:59,255][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:54:59,971][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:54:59,973][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:54:59,975][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:55:01,276][__main__][INFO] - Iteration 300 took 23s (38.55% Gen, 55.97% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 57m 43s. Estimated total time: 19h 48m 55s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 37s, 500 more iterations: 3h 18m 9s.
[2025-11-13 09:55:01,279][__main__][INFO] - Starting iteration 300.
[2025-11-13 09:55:01,282][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 29 and human policies 1.
[2025-11-13 09:55:01,282][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:55:09,754][__main__][INFO] - Number of regex retries in iteration 300: 0
[2025-11-13 09:55:09,755][__main__][INFO] - agents played in iteration 300 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:55:10,210][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:10,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:10,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:10,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:10,704][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:55:10,704][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:55:11,462][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:55:11,760][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:55:12,087][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:55:12,416][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:55:12,743][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:55:13,072][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:55:13,402][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:55:13,733][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:55:14,061][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:55:14,392][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:55:14,722][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:55:15,050][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:55:15,377][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:55:15,706][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:55:16,037][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:55:16,365][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:55:16,692][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:55:17,021][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:55:17,355][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:55:17,686][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:55:18,019][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:55:18,349][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:55:18,677][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:55:19,007][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:55:19,338][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:55:19,665][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:55:19,994][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:55:20,323][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:55:20,651][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:55:20,979][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:55:21,308][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:55:21,636][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:55:21,965][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:55:22,712][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:55:23,426][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:55:23,428][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:55:23,430][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:55:25,590][__main__][INFO] - Iteration 301 took 24s (34.85% Gen, 56.25% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 23m 50s. Estimated total time: 20h 15m 27s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 30s, 500 more iterations: 3h 22m 34s.
[2025-11-13 09:55:25,592][__main__][INFO] - Starting iteration 301.
[2025-11-13 09:55:25,595][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:55:25,596][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:55:35,011][__main__][INFO] - Number of regex retries in iteration 301: 0
[2025-11-13 09:55:35,012][__main__][INFO] - agents played in iteration 301 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:55:35,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:35,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:35,603][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:35,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:35,644][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:55:35,644][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:55:36,415][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:55:36,714][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:55:37,044][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:55:37,376][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:55:37,703][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:55:38,031][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:55:38,362][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:55:38,692][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:55:39,019][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:55:39,350][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:55:39,679][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:55:40,006][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:55:40,336][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:55:40,664][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:55:40,992][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:55:41,320][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:55:41,648][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:55:41,978][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:55:42,313][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:55:42,654][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:55:42,982][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:55:43,310][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:55:43,638][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:55:43,973][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:55:44,303][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:55:44,631][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:55:44,964][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:55:45,287][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:55:45,614][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:55:45,942][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:55:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:55:46,600][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:55:46,930][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:55:47,650][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:55:48,406][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:55:48,408][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:55:48,410][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:55:49,413][__main__][INFO] - Iteration 302 took 23s (39.53% Gen, 56.25% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 58m 54s. Estimated total time: 19h 50m 54s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 29s.
[2025-11-13 09:55:49,415][__main__][INFO] - Starting iteration 302.
[2025-11-13 09:55:49,419][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:55:49,419][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:55:58,596][__main__][INFO] - Number of regex retries in iteration 302: 0
[2025-11-13 09:55:58,597][__main__][INFO] - agents played in iteration 302 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:55:59,060][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:59,100][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:59,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:59,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:55:59,180][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:55:59,180][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:55:59,954][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:56:00,254][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:56:00,582][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:56:00,909][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:56:01,242][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:56:01,569][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:56:01,897][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:56:02,226][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:56:02,568][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:56:02,902][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:56:03,231][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:56:03,565][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:56:03,896][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:56:04,226][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:56:04,559][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:56:04,894][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:56:05,224][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:56:05,559][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:56:05,893][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:56:06,228][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:56:06,562][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:56:06,892][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:56:07,219][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:56:07,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:56:07,879][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:56:08,206][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:56:08,535][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:56:08,864][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:56:09,191][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:56:09,519][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:56:09,845][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:56:10,175][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:56:10,502][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:56:11,255][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:56:12,005][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:56:12,007][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:56:12,009][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:56:12,988][__main__][INFO] - Iteration 303 took 23s (38.94% Gen, 56.90% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 46m 7s. Estimated total time: 19h 38m 31s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 17s, 500 more iterations: 3h 16m 25s.
[2025-11-13 09:56:12,990][__main__][INFO] - Starting iteration 303.
[2025-11-13 09:56:12,993][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:56:12,994][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:56:21,902][__main__][INFO] - Number of regex retries in iteration 303: 0
[2025-11-13 09:56:21,903][__main__][INFO] - agents played in iteration 303 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:56:22,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:22,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:22,443][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:22,483][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:22,484][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:56:22,484][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:56:23,263][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:56:23,574][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:56:23,902][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:56:24,230][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:56:24,559][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:56:24,889][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:56:25,219][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:56:25,546][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:56:25,880][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:56:26,202][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:56:26,530][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:56:26,859][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:56:27,190][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:56:27,516][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:56:27,846][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:56:28,173][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:56:28,501][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:56:28,829][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:56:29,158][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:56:29,488][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:56:29,818][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:56:30,156][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:56:30,484][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:56:30,813][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:56:31,144][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:56:31,474][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:56:31,801][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:56:32,129][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:56:32,459][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:56:32,788][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:56:33,116][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:56:33,444][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:56:33,777][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:56:34,539][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:56:35,285][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:56:35,286][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:56:35,288][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:56:36,275][__main__][INFO] - Iteration 304 took 23s (38.26% Gen, 57.49% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 31m 20s. Estimated total time: 19h 24m 7s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 1s.
[2025-11-13 09:56:36,277][__main__][INFO] - Starting iteration 304.
[2025-11-13 09:56:36,281][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:56:36,282][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:56:45,462][__main__][INFO] - Number of regex retries in iteration 304: 0
[2025-11-13 09:56:45,462][__main__][INFO] - agents played in iteration 304 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:56:45,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:45,982][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:46,022][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:46,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:56:46,063][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:56:46,063][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:56:46,823][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:56:47,121][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:56:47,455][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:56:47,782][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:56:48,110][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:56:48,439][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:56:48,777][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:56:49,104][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:56:49,432][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:56:49,766][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:56:50,095][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:56:50,423][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:56:50,751][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:56:51,084][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:56:51,408][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:56:51,736][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:56:52,065][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:56:52,399][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:56:52,730][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:56:53,059][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:56:53,390][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:56:53,724][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:56:54,054][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:56:54,382][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:56:54,709][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:56:55,038][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:56:55,369][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:56:55,697][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:56:56,025][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:56:56,353][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:56:56,680][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:56:57,008][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:56:57,336][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:56:58,087][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:56:58,791][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:56:58,793][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:56:58,795][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:56:59,794][__main__][INFO] - Iteration 305 took 23s (39.04% Gen, 56.70% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 42m 32s. Estimated total time: 19h 35m 43s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 57s.
[2025-11-13 09:56:59,797][__main__][INFO] - Starting iteration 305.
[2025-11-13 09:56:59,800][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:56:59,801][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:57:08,456][__main__][INFO] - Number of regex retries in iteration 305: 0
[2025-11-13 09:57:08,457][__main__][INFO] - agents played in iteration 305 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:57:08,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:08,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:08,991][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:09,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:09,032][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:57:09,033][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:57:09,818][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:57:10,115][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:57:10,446][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:57:10,773][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:57:11,102][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:57:11,430][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:57:11,758][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:57:12,087][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:57:12,422][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:57:12,752][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:57:13,078][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:57:13,409][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:57:13,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:57:14,070][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:57:14,402][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:57:14,729][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:57:15,058][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:57:15,388][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:57:15,718][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:57:16,046][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:57:16,380][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:57:16,708][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:57:17,039][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:57:17,367][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:57:17,695][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:57:18,023][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:57:18,351][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:57:18,679][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:57:19,007][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:57:19,334][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:57:19,662][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:57:19,991][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:57:20,320][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:57:21,059][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:57:21,791][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:57:21,792][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:57:21,794][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:57:22,816][__main__][INFO] - Iteration 306 took 23s (37.61% Gen, 57.94% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 17m 17s. Estimated total time: 19h 10m 51s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 21s, 500 more iterations: 3h 11m 48s.
[2025-11-13 09:57:22,819][__main__][INFO] - Starting iteration 306.
[2025-11-13 09:57:22,822][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:57:22,823][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:57:30,806][__main__][INFO] - Number of regex retries in iteration 306: 0
[2025-11-13 09:57:30,807][__main__][INFO] - agents played in iteration 306 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:57:31,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:31,309][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:31,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:31,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:31,390][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:57:31,390][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:57:32,147][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:57:32,445][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:57:32,772][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:57:33,100][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:57:33,426][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:57:33,753][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:57:34,081][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:57:34,409][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:57:34,737][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:57:35,064][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:57:35,392][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:57:35,723][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:57:36,053][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:57:36,381][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:57:36,709][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:57:37,036][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:57:37,366][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:57:37,694][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:57:38,024][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:57:38,352][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:57:38,690][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:57:39,022][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:57:39,353][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:57:39,684][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:57:40,016][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:57:40,344][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:57:40,673][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:57:41,004][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:57:41,331][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:57:41,661][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:57:41,992][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:57:42,321][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:57:42,649][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:57:43,374][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:57:44,097][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:57:44,098][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:57:44,099][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:57:45,053][__main__][INFO] - Iteration 307 took 22s (35.91% Gen, 59.79% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 37m 41s. Estimated total time: 18h 31m 37s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 3s, 500 more iterations: 3h 5m 16s.
[2025-11-13 09:57:45,055][__main__][INFO] - Starting iteration 307.
[2025-11-13 09:57:45,058][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:57:45,059][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:57:53,647][__main__][INFO] - Number of regex retries in iteration 307: 0
[2025-11-13 09:57:53,647][__main__][INFO] - agents played in iteration 307 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:57:54,103][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:54,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:54,549][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:54,589][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:57:54,590][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:57:54,590][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:57:55,351][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:57:55,649][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:57:55,977][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:57:56,304][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:57:56,633][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:57:56,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:57:57,295][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:57:57,623][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:57:57,951][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:57:58,280][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:57:58,607][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:57:58,935][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:57:59,261][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:57:59,590][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:57:59,918][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:58:00,246][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:58:00,573][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:58:00,900][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:58:01,231][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:58:01,559][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:58:01,893][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:58:02,221][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:58:02,549][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:58:02,880][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:58:03,208][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:58:03,537][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:58:03,867][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:58:04,196][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:58:04,525][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:58:04,852][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:58:05,180][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:58:05,507][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:58:05,835][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:58:06,586][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:58:07,309][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:58:07,311][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:58:07,313][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:58:08,306][__main__][INFO] - Iteration 308 took 23s (36.94% Gen, 58.78% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 28m 7s. Estimated total time: 19h 22m 26s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 44s, 500 more iterations: 3h 13m 44s.
[2025-11-13 09:58:08,308][__main__][INFO] - Starting iteration 308.
[2025-11-13 09:58:08,312][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:58:08,312][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:58:17,384][__main__][INFO] - Number of regex retries in iteration 308: 0
[2025-11-13 09:58:17,385][__main__][INFO] - agents played in iteration 308 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:58:17,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:17,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:17,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:17,959][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:17,960][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:58:17,961][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:58:18,729][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:58:19,027][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:58:19,355][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:58:19,682][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:58:20,010][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:58:20,337][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:58:20,664][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:58:20,992][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:58:21,320][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:58:21,648][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:58:21,990][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:58:22,319][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:58:22,647][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:58:22,975][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:58:23,302][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:58:23,629][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:58:23,957][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:58:24,287][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:58:24,615][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:58:24,943][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:58:25,272][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:58:25,602][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:58:25,936][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:58:26,264][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:58:26,595][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:58:26,929][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:58:27,260][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:58:27,588][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:58:27,916][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:58:28,246][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:58:28,579][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:58:28,907][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:58:29,234][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:58:29,962][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:58:30,663][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:58:30,665][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:58:30,667][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:58:31,811][__main__][INFO] - Iteration 309 took 23s (38.61% Gen, 56.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 40m 17s. Estimated total time: 19h 35m 0s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 50s.
[2025-11-13 09:58:31,813][__main__][INFO] - Starting iteration 309.
[2025-11-13 09:58:31,816][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:58:31,817][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:58:40,252][__main__][INFO] - Number of regex retries in iteration 309: 0
[2025-11-13 09:58:40,252][__main__][INFO] - agents played in iteration 309 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:58:40,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,763][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,803][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,844][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:58:40,845][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:58:40,845][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:58:41,637][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:58:41,935][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:58:42,262][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:58:42,596][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:58:42,917][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:58:43,244][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:58:43,573][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:58:43,901][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:58:44,228][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:58:44,556][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:58:44,884][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:58:45,213][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:58:45,542][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:58:45,875][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:58:46,209][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:58:46,535][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:58:46,864][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:58:47,191][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:58:47,519][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:58:47,848][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:58:48,182][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:58:48,509][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:58:48,838][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:58:49,169][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:58:49,502][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:58:49,837][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:58:50,165][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:58:50,493][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:58:50,820][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:58:51,148][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:58:51,479][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:58:51,806][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:58:52,134][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:58:52,851][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:58:53,556][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:58:53,557][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:58:53,559][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:58:54,782][__main__][INFO] - Iteration 310 took 22s (36.73% Gen, 57.94% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 13m 14s. Estimated total time: 19h 8m 20s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 16s, 500 more iterations: 3h 11m 23s.
[2025-11-13 09:58:54,784][__main__][INFO] - Starting iteration 310.
[2025-11-13 09:58:54,787][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 30 and human policies 1.
[2025-11-13 09:58:54,788][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:59:03,619][__main__][INFO] - Number of regex retries in iteration 310: 0
[2025-11-13 09:59:03,620][__main__][INFO] - agents played in iteration 310 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:59:04,085][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:04,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:04,165][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:04,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:04,207][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:59:04,208][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:59:05,006][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:59:05,304][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:59:05,632][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:59:05,959][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:59:06,293][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:59:06,614][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:59:06,943][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:59:07,271][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:59:07,605][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:59:07,925][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:59:08,252][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:59:08,579][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:59:08,906][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:59:09,234][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:59:09,562][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:59:09,890][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:59:10,222][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:59:10,563][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:59:10,899][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:59:11,230][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:59:11,561][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:59:11,894][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:59:12,222][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:59:12,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:59:12,883][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:59:13,208][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:59:13,538][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:59:13,866][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:59:14,193][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:59:14,521][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:59:14,849][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:59:15,176][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:59:15,504][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:59:16,224][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:59:16,946][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:59:16,948][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:59:16,950][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:59:19,017][__main__][INFO] - Iteration 311 took 24s (36.45% Gen, 55.01% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 16m 4s. Estimated total time: 20h 11m 34s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 23s, 500 more iterations: 3h 21m 55s.
[2025-11-13 09:59:19,019][__main__][INFO] - Starting iteration 311.
[2025-11-13 09:59:19,023][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 09:59:19,024][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:59:27,354][__main__][INFO] - Number of regex retries in iteration 311: 0
[2025-11-13 09:59:27,355][__main__][INFO] - agents played in iteration 311 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:59:27,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:27,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:27,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:27,944][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:27,944][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:59:27,945][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:59:28,725][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:59:29,023][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:59:29,350][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:59:29,679][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:59:30,006][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:59:30,334][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:59:30,663][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:59:30,991][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:59:31,319][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:59:31,647][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:59:31,976][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:59:32,304][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:59:32,632][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:59:32,960][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:59:33,288][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:59:33,616][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:59:33,951][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:59:34,290][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:59:34,617][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:59:34,946][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:59:35,277][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:59:35,606][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:59:35,940][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:59:36,269][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 09:59:36,603][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 09:59:36,928][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 09:59:37,256][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 09:59:37,585][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 09:59:37,913][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 09:59:38,243][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 09:59:38,571][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 09:59:38,899][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 09:59:39,226][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 09:59:39,959][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 09:59:40,691][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 09:59:40,693][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 09:59:40,694][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 09:59:41,655][__main__][INFO] - Iteration 312 took 22s (36.81% Gen, 58.94% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 55m 45s. Estimated total time: 18h 51m 38s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 43s, 500 more iterations: 3h 8m 36s.
[2025-11-13 09:59:41,657][__main__][INFO] - Starting iteration 312.
[2025-11-13 09:59:41,660][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 09:59:41,661][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 09:59:50,939][__main__][INFO] - Number of regex retries in iteration 312: 0
[2025-11-13 09:59:50,940][__main__][INFO] - agents played in iteration 312 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 09:59:51,394][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:51,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:51,476][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:51,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 09:59:51,517][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 09:59:51,517][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 09:59:52,307][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 09:59:52,605][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 09:59:52,932][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 09:59:53,261][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 09:59:53,594][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 09:59:53,917][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 09:59:54,245][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 09:59:54,572][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 09:59:54,905][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 09:59:55,228][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 09:59:55,556][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 09:59:55,885][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 09:59:56,214][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 09:59:56,547][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 09:59:56,875][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 09:59:57,204][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 09:59:57,534][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 09:59:57,862][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 09:59:58,193][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 09:59:58,524][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 09:59:58,852][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 09:59:59,183][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 09:59:59,511][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 09:59:59,840][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:00:00,170][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:00:00,499][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:00:00,827][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:00:01,155][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:00:01,484][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:00:01,813][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:00:02,140][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:00:02,474][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:00:02,801][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:00:03,536][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:00:04,261][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:00:04,262][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:00:04,264][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:00:05,191][__main__][INFO] - Iteration 313 took 23s (39.44% Gen, 56.62% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 40m 20s. Estimated total time: 19h 36m 37s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 13s, 500 more iterations: 3h 16m 6s.
[2025-11-13 10:00:05,193][__main__][INFO] - Starting iteration 313.
[2025-11-13 10:00:05,197][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:00:05,198][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:00:14,315][__main__][INFO] - Number of regex retries in iteration 313: 0
[2025-11-13 10:00:14,315][__main__][INFO] - agents played in iteration 313 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:00:14,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:14,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:14,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:14,904][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:14,904][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:00:14,905][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:00:15,681][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:00:15,979][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:00:16,308][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:00:16,634][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:00:16,964][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:00:17,291][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:00:17,622][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:00:17,949][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:00:18,277][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:00:18,605][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:00:18,934][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:00:19,262][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:00:19,590][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:00:19,919][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:00:20,246][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:00:20,574][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:00:20,903][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:00:21,232][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:00:21,562][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:00:21,895][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:00:22,226][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:00:22,561][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:00:22,899][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:00:23,231][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:00:23,559][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:00:23,886][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:00:24,215][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:00:24,543][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:00:24,872][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:00:25,199][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:00:25,529][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:00:25,857][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:00:26,185][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:00:26,920][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:00:27,621][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:00:27,623][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:00:27,625][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:00:28,564][__main__][INFO] - Iteration 314 took 23s (39.02% Gen, 56.96% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 31m 44s. Estimated total time: 19h 28m 23s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 43s.
[2025-11-13 10:00:28,566][__main__][INFO] - Starting iteration 314.
[2025-11-13 10:00:28,570][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:00:28,570][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:00:37,862][__main__][INFO] - Number of regex retries in iteration 314: 0
[2025-11-13 10:00:37,863][__main__][INFO] - agents played in iteration 314 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:00:38,318][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:38,358][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:38,398][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:38,439][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:00:38,439][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:00:38,439][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:00:39,203][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:00:39,500][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:00:39,829][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:00:40,155][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:00:40,483][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:00:40,810][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:00:41,140][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:00:41,468][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:00:41,796][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:00:42,124][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:00:42,451][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:00:42,778][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:00:43,105][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:00:43,432][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:00:43,760][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:00:44,088][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:00:44,420][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:00:44,749][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:00:45,080][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:00:45,409][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:00:45,742][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:00:46,071][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:00:46,400][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:00:46,728][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:00:47,056][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:00:47,384][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:00:47,714][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:00:48,044][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:00:48,372][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:00:48,702][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:00:49,031][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:00:49,359][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:00:49,686][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:00:50,443][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:00:51,147][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:00:51,149][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:00:51,150][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:00:52,425][__main__][INFO] - Iteration 315 took 23s (38.95% Gen, 55.70% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 55m 45s. Estimated total time: 19h 52m 49s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 45s, 500 more iterations: 3h 18m 48s.
[2025-11-13 10:00:52,427][__main__][INFO] - Starting iteration 315.
[2025-11-13 10:00:52,431][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:00:52,432][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:01,041][__main__][INFO] - Number of regex retries in iteration 315: 0
[2025-11-13 10:01:01,041][__main__][INFO] - agents played in iteration 315 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:01:01,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:01,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:01,580][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:01,620][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:01,621][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:01:01,621][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:01:02,426][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:01:02,801][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:01:03,128][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:01:03,456][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:01:03,782][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:01:04,110][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:01:04,438][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:01:04,767][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:01:05,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:01:05,433][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:01:05,761][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:01:06,090][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:01:06,421][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:01:06,748][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:01:07,077][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:01:07,410][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:01:07,732][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:01:08,060][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:01:08,388][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:01:08,717][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:01:09,050][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:01:09,378][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:01:09,706][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:01:10,040][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:01:10,370][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:01:10,699][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:01:11,028][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:01:11,356][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:01:11,685][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:01:12,014][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:01:12,342][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:01:12,672][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:01:13,004][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:01:13,779][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:01:14,512][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:01:14,513][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:01:14,515][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:01:15,432][__main__][INFO] - Iteration 316 took 23s (37.43% Gen, 58.58% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 12m 37s. Estimated total time: 19h 10m 3s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 20s, 500 more iterations: 3h 11m 40s.
[2025-11-13 10:01:15,434][__main__][INFO] - Starting iteration 316.
[2025-11-13 10:01:15,437][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:01:15,438][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:24,244][__main__][INFO] - Number of regex retries in iteration 316: 0
[2025-11-13 10:01:24,245][__main__][INFO] - agents played in iteration 316 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:01:24,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:24,747][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:24,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:24,828][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:24,829][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:01:24,829][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:01:25,604][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:01:25,903][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:01:26,230][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:01:26,558][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:01:26,887][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:01:27,214][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:01:27,542][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:01:27,870][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:01:28,197][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:01:28,531][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:01:28,855][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:01:29,182][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:01:29,510][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:01:29,838][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:01:30,165][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:01:30,492][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:01:30,820][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:01:31,147][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:01:31,474][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:01:31,802][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:01:32,129][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:01:32,457][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:01:32,785][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:01:33,113][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:01:33,442][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:01:33,774][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:01:34,101][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:01:34,431][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:01:34,760][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:01:35,086][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:01:35,413][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:01:35,741][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:01:36,070][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:01:36,843][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:01:37,554][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:01:37,556][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:01:37,557][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:01:38,473][__main__][INFO] - Iteration 317 took 23s (38.23% Gen, 57.79% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 14m 0s. Estimated total time: 19h 11m 50s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 58s.
[2025-11-13 10:01:38,475][__main__][INFO] - Starting iteration 317.
[2025-11-13 10:01:38,478][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:01:38,479][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:01:47,418][__main__][INFO] - Number of regex retries in iteration 317: 0
[2025-11-13 10:01:47,419][__main__][INFO] - agents played in iteration 317 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:01:47,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:47,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:47,952][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:47,993][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:01:47,993][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:01:47,994][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:01:48,778][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:01:49,076][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:01:49,405][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:01:49,738][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:01:50,069][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:01:50,396][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:01:50,723][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:01:51,051][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:01:51,379][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:01:51,706][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:01:52,033][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:01:52,360][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:01:52,688][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:01:53,016][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:01:53,344][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:01:53,673][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:01:54,000][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:01:54,328][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:01:54,656][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:01:54,984][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:01:55,313][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:01:55,642][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:01:55,969][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:01:56,303][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:01:56,633][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:01:56,960][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:01:57,292][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:01:57,624][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:01:57,952][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:01:58,280][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:01:58,607][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:01:58,937][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:01:59,264][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:02:00,023][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:02:00,730][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:02:00,731][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:02:00,732][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:02:01,666][__main__][INFO] - Iteration 318 took 23s (38.55% Gen, 57.42% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 21m 13s. Estimated total time: 19h 19m 26s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 14s.
[2025-11-13 10:02:01,668][__main__][INFO] - Starting iteration 318.
[2025-11-13 10:02:01,672][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:02:01,672][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:02:10,803][__main__][INFO] - Number of regex retries in iteration 318: 0
[2025-11-13 10:02:10,804][__main__][INFO] - agents played in iteration 318 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:02:11,253][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:11,292][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:11,331][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:11,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:11,371][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:02:11,372][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:02:12,146][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:02:12,444][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:02:12,776][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:02:13,104][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:02:13,435][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:02:13,770][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:02:14,099][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:02:14,427][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:02:14,754][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:02:15,081][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:02:15,408][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:02:15,741][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:02:16,064][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:02:16,391][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:02:16,719][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:02:17,047][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:02:17,374][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:02:17,701][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:02:18,027][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:02:18,354][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:02:18,682][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:02:19,010][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:02:19,338][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:02:19,667][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:02:20,000][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:02:20,328][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:02:20,659][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:02:20,987][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:02:21,316][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:02:21,644][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:02:21,972][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:02:22,300][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:02:22,628][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:02:23,381][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:02:24,122][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:02:24,124][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:02:24,125][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:02:25,341][__main__][INFO] - Iteration 319 took 23s (38.58% Gen, 56.28% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 44m 55s. Estimated total time: 19h 43m 31s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 27s, 500 more iterations: 3h 17m 15s.
[2025-11-13 10:02:25,344][__main__][INFO] - Starting iteration 319.
[2025-11-13 10:02:25,347][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:02:25,348][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:02:33,998][__main__][INFO] - Number of regex retries in iteration 319: 0
[2025-11-13 10:02:33,998][__main__][INFO] - agents played in iteration 319 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:02:34,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:34,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:34,511][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:34,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:34,551][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:02:34,551][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:02:35,263][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:02:35,559][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:02:35,888][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:02:36,215][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:02:36,549][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:02:36,886][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:02:37,213][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:02:37,546][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:02:37,876][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:02:38,211][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:02:38,538][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:02:38,865][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:02:39,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:02:39,535][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:02:39,862][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:02:40,190][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:02:40,517][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:02:40,848][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:02:41,176][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:02:41,504][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:02:41,838][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:02:42,159][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:02:42,490][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:02:42,818][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:02:43,145][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:02:43,478][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:02:43,807][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:02:44,134][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:02:44,462][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:02:44,792][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:02:45,119][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:02:45,446][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:02:45,776][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:02:46,523][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:02:47,233][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:02:47,236][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:02:47,237][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:02:48,216][__main__][INFO] - Iteration 320 took 22s (37.82% Gen, 57.89% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 4m 32s. Estimated total time: 19h 3m 31s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 7s, 500 more iterations: 3h 10m 35s.
[2025-11-13 10:02:48,218][__main__][INFO] - Starting iteration 320.
[2025-11-13 10:02:48,221][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 31 and human policies 1.
[2025-11-13 10:02:48,222][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:02:57,147][__main__][INFO] - Number of regex retries in iteration 320: 0
[2025-11-13 10:02:57,148][__main__][INFO] - agents played in iteration 320 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:02:57,584][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:57,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:57,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:57,721][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:02:57,722][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:02:57,722][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:02:58,438][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:02:58,735][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:02:59,065][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:02:59,395][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:02:59,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:03:00,058][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:03:00,389][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:03:00,717][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:03:01,045][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:03:01,375][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:03:01,709][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:03:02,040][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:03:02,367][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:03:02,696][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:03:03,027][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:03:03,354][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:03:03,682][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:03:04,010][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:03:04,337][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:03:04,666][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:03:04,993][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:03:05,321][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:03:05,653][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:03:05,979][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:03:06,306][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:03:06,634][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:03:06,964][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:03:07,292][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:03:07,620][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:03:07,948][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:03:08,278][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:03:08,611][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:03:08,939][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:03:09,712][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:03:10,420][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:10,422][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:10,424][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:12,254][__main__][INFO] - Iteration 321 took 24s (37.14% Gen, 55.24% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 2m 19s. Estimated total time: 20h 1m 43s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 17s.
[2025-11-13 10:03:12,256][__main__][INFO] - Starting iteration 321.
[2025-11-13 10:03:12,260][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:03:12,260][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:03:21,057][__main__][INFO] - Number of regex retries in iteration 321: 0
[2025-11-13 10:03:21,058][__main__][INFO] - agents played in iteration 321 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:03:21,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:21,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:21,584][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:21,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:21,624][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:03:21,624][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:03:22,336][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:03:22,633][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:03:22,962][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:03:23,289][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:03:23,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:03:23,947][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:03:24,273][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:03:24,600][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:03:24,933][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:03:25,264][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:03:25,605][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:03:25,939][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:03:26,269][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:03:26,596][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:03:26,937][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:03:27,266][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:03:27,592][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:03:27,919][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:03:28,252][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:03:28,579][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:03:28,907][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:03:29,240][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:03:29,564][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:03:29,892][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:03:30,220][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:03:30,548][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:03:30,875][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:03:31,203][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:03:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:03:31,859][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:03:32,186][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:03:32,513][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:03:32,841][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:03:33,605][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:03:34,325][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:34,327][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:34,328][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:35,277][__main__][INFO] - Iteration 322 took 23s (38.21% Gen, 57.65% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 11m 7s. Estimated total time: 19h 10m 54s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 21s, 500 more iterations: 3h 11m 49s.
[2025-11-13 10:03:35,280][__main__][INFO] - Starting iteration 322.
[2025-11-13 10:03:35,283][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:03:35,284][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:03:44,319][__main__][INFO] - Number of regex retries in iteration 322: 0
[2025-11-13 10:03:44,319][__main__][INFO] - agents played in iteration 322 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:03:44,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:44,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:44,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:44,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:03:44,873][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:03:44,873][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:03:45,590][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:03:45,886][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:03:46,213][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:03:46,551][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:03:46,874][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:03:47,202][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:03:47,530][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:03:47,862][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:03:48,195][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:03:48,525][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:03:48,858][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:03:49,187][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:03:49,530][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:03:49,860][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:03:50,193][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:03:50,522][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:03:50,854][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:03:51,181][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:03:51,509][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:03:51,837][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:03:52,164][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:03:52,493][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:03:52,820][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:03:53,147][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:03:53,477][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:03:53,805][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:03:54,133][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:03:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:03:54,789][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:03:55,118][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:03:55,445][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:03:55,771][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:03:56,101][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:03:56,869][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:03:57,594][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:03:57,597][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:03:57,598][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:03:58,524][__main__][INFO] - Iteration 323 took 23s (38.87% Gen, 57.13% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 21m 57s. Estimated total time: 19h 22m 6s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 44s, 500 more iterations: 3h 13m 41s.
[2025-11-13 10:03:58,527][__main__][INFO] - Starting iteration 323.
[2025-11-13 10:03:58,529][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:03:58,530][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:07,370][__main__][INFO] - Number of regex retries in iteration 323: 0
[2025-11-13 10:04:07,371][__main__][INFO] - agents played in iteration 323 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:04:07,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:07,855][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:07,894][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:07,934][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:07,934][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:07,935][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:08,653][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:04:08,950][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:04:09,277][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:04:09,604][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:04:09,930][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:04:10,256][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:04:10,583][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:04:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:04:11,238][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:04:11,568][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:04:11,895][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:04:12,223][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:04:12,552][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:04:12,879][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:04:13,211][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:04:13,541][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:04:13,867][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:04:14,196][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:04:14,530][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:04:14,857][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:04:15,185][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:04:15,513][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:04:15,841][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:04:16,168][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:04:16,495][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:04:16,824][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:04:17,156][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:04:17,482][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:04:17,810][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:04:18,141][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:04:18,469][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:04:18,798][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:04:19,126][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:04:19,892][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:04:20,592][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:04:20,594][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:04:20,596][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:04:21,533][__main__][INFO] - Iteration 324 took 23s (38.43% Gen, 57.49% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 9m 41s. Estimated total time: 19h 10m 14s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 20s, 500 more iterations: 3h 11m 42s.
[2025-11-13 10:04:21,535][__main__][INFO] - Starting iteration 324.
[2025-11-13 10:04:21,538][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:04:21,539][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:30,400][__main__][INFO] - Number of regex retries in iteration 324: 0
[2025-11-13 10:04:30,400][__main__][INFO] - agents played in iteration 324 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:04:30,846][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:30,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:30,924][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:30,964][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:30,964][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:30,964][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:31,700][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:04:31,998][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:04:32,326][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:04:32,662][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:04:32,986][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:04:33,315][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:04:33,644][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:04:33,975][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:04:34,297][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:04:34,623][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:04:34,952][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:04:35,286][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:04:35,618][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:04:35,946][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:04:36,273][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:04:36,601][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:04:36,934][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:04:37,262][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:04:37,596][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:04:37,927][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:04:38,265][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:04:38,594][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:04:38,921][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:04:39,249][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:04:39,577][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:04:39,905][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:04:40,234][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:04:40,567][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:04:40,890][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:04:41,218][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:04:41,546][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:04:41,878][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:04:42,201][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:04:42,960][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:04:43,681][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:04:43,683][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:04:43,684][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:04:44,629][__main__][INFO] - Iteration 325 took 23s (38.37% Gen, 57.53% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 13m 40s. Estimated total time: 19h 14m 36s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 29s, 500 more iterations: 3h 12m 26s.
[2025-11-13 10:04:44,631][__main__][INFO] - Starting iteration 325.
[2025-11-13 10:04:44,634][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:04:44,635][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:04:53,735][__main__][INFO] - Number of regex retries in iteration 325: 0
[2025-11-13 10:04:53,736][__main__][INFO] - agents played in iteration 325 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:04:54,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:54,221][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:54,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:54,299][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:04:54,300][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:04:54,300][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:04:55,012][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:04:55,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:04:55,638][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:04:55,965][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:04:56,292][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:04:56,618][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:04:56,943][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:04:57,269][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:04:57,598][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:04:57,927][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:04:58,252][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:04:58,578][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:04:58,905][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:04:59,234][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:04:59,564][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:04:59,891][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:05:00,219][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:05:00,546][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:05:00,879][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:05:01,208][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:05:01,535][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:05:01,863][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:05:02,190][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:05:02,517][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:05:02,845][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:05:03,174][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:05:03,501][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:05:03,828][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:05:04,160][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:05:04,483][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:05:04,811][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:05:05,139][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:05:05,467][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:05:06,234][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:05:06,979][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:06,980][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:06,982][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:07,886][__main__][INFO] - Iteration 326 took 23s (39.14% Gen, 56.97% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 21m 20s. Estimated total time: 19h 22m 39s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 45s, 500 more iterations: 3h 13m 46s.
[2025-11-13 10:05:07,888][__main__][INFO] - Starting iteration 326.
[2025-11-13 10:05:07,891][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:05:07,892][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:05:17,231][__main__][INFO] - Number of regex retries in iteration 326: 0
[2025-11-13 10:05:17,232][__main__][INFO] - agents played in iteration 326 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:05:17,685][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:17,725][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:17,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:17,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:17,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:05:17,807][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:05:18,519][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:05:18,816][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:05:19,144][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:05:19,473][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:05:19,801][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:05:20,128][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:05:20,468][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:05:20,795][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:05:21,122][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:05:21,455][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:05:21,785][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:05:22,115][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:05:22,443][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:05:22,770][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:05:23,097][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:05:23,424][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:05:23,755][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:05:24,082][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:05:24,410][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:05:24,743][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:05:25,071][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:05:25,399][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:05:25,727][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:05:26,056][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:05:26,382][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:05:26,709][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:05:27,035][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:05:27,363][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:05:27,690][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:05:28,018][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:05:28,346][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:05:28,676][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:05:29,005][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:05:29,787][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:05:30,513][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:30,514][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:30,515][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:31,442][__main__][INFO] - Iteration 327 took 23s (39.66% Gen, 56.40% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 35m 52s. Estimated total time: 19h 37m 34s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 15s, 500 more iterations: 3h 16m 15s.
[2025-11-13 10:05:31,444][__main__][INFO] - Starting iteration 327.
[2025-11-13 10:05:31,447][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:05:31,447][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:05:41,244][__main__][INFO] - Number of regex retries in iteration 327: 0
[2025-11-13 10:05:41,245][__main__][INFO] - agents played in iteration 327 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:05:41,678][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:41,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:41,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:41,796][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:05:41,797][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:05:41,797][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:05:42,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:05:42,815][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:05:43,142][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:05:43,469][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:05:43,802][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:05:44,132][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:05:44,462][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:05:44,791][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:05:45,126][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:05:45,455][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:05:45,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:05:46,110][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:05:46,437][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:05:46,766][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:05:47,093][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:05:47,421][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:05:47,755][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:05:48,083][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:05:48,416][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:05:48,749][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:05:49,087][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:05:49,418][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:05:49,747][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:05:50,083][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:05:50,403][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:05:50,730][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:05:51,058][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:05:51,390][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:05:51,713][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:05:52,041][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:05:52,369][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:05:52,697][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:05:53,024][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:05:53,804][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:05:54,524][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:05:54,526][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:05:54,527][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:05:55,481][__main__][INFO] - Iteration 328 took 24s (40.76% Gen, 55.27% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 59m 39s. Estimated total time: 20h 1m 46s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 17s.
[2025-11-13 10:05:55,483][__main__][INFO] - Starting iteration 328.
[2025-11-13 10:05:55,486][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:05:55,487][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:06:04,889][__main__][INFO] - Number of regex retries in iteration 328: 0
[2025-11-13 10:06:04,890][__main__][INFO] - agents played in iteration 328 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:06:05,321][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:05,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:05,402][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:05,442][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:05,442][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:06:05,442][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:06:06,163][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:06:06,460][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:06:06,788][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:06:07,114][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:06:07,442][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:06:07,772][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:06:08,099][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:06:08,426][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:06:08,754][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:06:09,083][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:06:09,412][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:06:09,739][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:06:10,066][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:06:10,393][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:06:10,720][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:06:11,055][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:06:11,383][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:06:11,709][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:06:12,036][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:06:12,365][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:06:12,698][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:06:13,021][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:06:13,349][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:06:13,677][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:06:14,005][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:06:14,334][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:06:14,662][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:06:14,990][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:06:15,319][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:06:15,648][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:06:15,975][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:06:16,304][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:06:16,632][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:06:17,405][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:06:18,108][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:06:18,110][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:06:18,112][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:06:19,312][__main__][INFO] - Iteration 329 took 23s (39.46% Gen, 55.49% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 48m 50s. Estimated total time: 19h 51m 20s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 42s, 500 more iterations: 3h 18m 33s.
[2025-11-13 10:06:19,314][__main__][INFO] - Starting iteration 329.
[2025-11-13 10:06:19,317][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:06:19,317][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:06:28,607][__main__][INFO] - Number of regex retries in iteration 329: 0
[2025-11-13 10:06:28,607][__main__][INFO] - agents played in iteration 329 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:06:29,055][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:29,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:29,133][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:29,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:29,173][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:06:29,173][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:06:29,903][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:06:30,199][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:06:30,531][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:06:30,857][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:06:31,185][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:06:31,514][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:06:31,841][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:06:32,170][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:06:32,499][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:06:32,829][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:06:33,161][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:06:33,493][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:06:33,824][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:06:34,154][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:06:34,487][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:06:34,817][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:06:35,141][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:06:35,468][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:06:35,795][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:06:36,122][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:06:36,451][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:06:36,779][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:06:37,110][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:06:37,437][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:06:37,764][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:06:38,092][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:06:38,420][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:06:38,747][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:06:39,083][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:06:39,411][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:06:39,738][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:06:40,067][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:06:40,394][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:06:41,164][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:06:41,871][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:06:41,873][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:06:41,874][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:06:42,827][__main__][INFO] - Iteration 330 took 23s (39.51% Gen, 56.43% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 32m 38s. Estimated total time: 19h 35m 32s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 11s, 500 more iterations: 3h 15m 55s.
[2025-11-13 10:06:42,829][__main__][INFO] - Starting iteration 330.
[2025-11-13 10:06:42,832][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 32 and human policies 1.
[2025-11-13 10:06:42,832][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:06:52,263][__main__][INFO] - Number of regex retries in iteration 330: 0
[2025-11-13 10:06:52,264][__main__][INFO] - agents played in iteration 330 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:06:52,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:52,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:52,794][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:52,837][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:06:52,838][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:06:52,838][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:06:53,607][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:06:53,904][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:06:54,232][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:06:54,558][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:06:54,894][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:06:55,223][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:06:55,555][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:06:55,882][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:06:56,214][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:06:56,541][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:06:56,870][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:06:57,203][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:06:57,532][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:06:57,866][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:06:58,199][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:06:58,533][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:06:58,863][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:06:59,200][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:06:59,533][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:06:59,866][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:07:00,196][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:07:00,526][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:07:00,856][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:07:01,187][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:07:01,521][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:07:01,851][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:07:02,178][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:07:02,508][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:07:02,835][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:07:03,163][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:07:03,490][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:07:03,818][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:07:04,147][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:07:04,908][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:07:05,635][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:07:05,636][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:07:05,638][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:07,402][__main__][INFO] - Iteration 331 took 24s (38.38% Gen, 54.43% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 25m 15s. Estimated total time: 20h 28m 33s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 57s, 500 more iterations: 3h 24m 45s.
[2025-11-13 10:07:07,404][__main__][INFO] - Starting iteration 331.
[2025-11-13 10:07:07,407][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:07:07,408][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:07:16,365][__main__][INFO] - Number of regex retries in iteration 331: 0
[2025-11-13 10:07:16,366][__main__][INFO] - agents played in iteration 331 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:07:16,815][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:16,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:16,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:16,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:16,937][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:07:16,937][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:07:17,683][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:07:17,979][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:07:18,306][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:07:18,635][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:07:18,961][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:07:19,287][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:07:19,614][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:07:19,941][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:07:20,267][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:07:20,595][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:07:20,922][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:07:21,249][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:07:21,576][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:07:21,902][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:07:22,229][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:07:22,559][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:07:22,888][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:07:23,222][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:07:23,547][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:07:23,872][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:07:24,201][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:07:24,531][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:07:24,864][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:07:25,195][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:07:25,522][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:07:25,849][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:07:26,185][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:07:26,513][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:07:26,841][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:07:27,169][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:07:27,507][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:07:27,835][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:07:28,163][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:07:28,957][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:07:29,672][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:07:29,674][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:07:29,676][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:30,596][__main__][INFO] - Iteration 332 took 23s (38.63% Gen, 57.40% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 15m 47s. Estimated total time: 19h 19m 29s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 14s.
[2025-11-13 10:07:30,598][__main__][INFO] - Starting iteration 332.
[2025-11-13 10:07:30,601][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:07:30,601][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:07:39,494][__main__][INFO] - Number of regex retries in iteration 332: 0
[2025-11-13 10:07:39,495][__main__][INFO] - agents played in iteration 332 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:07:39,948][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:39,989][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:40,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:40,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:07:40,069][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:07:40,069][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:07:40,847][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:07:41,145][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:07:41,476][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:07:41,806][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:07:42,133][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:07:42,462][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:07:42,791][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:07:43,129][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:07:43,460][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:07:43,788][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:07:44,119][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:07:44,448][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:07:44,778][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:07:45,111][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:07:45,445][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:07:45,780][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:07:46,110][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:07:46,436][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:07:46,770][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:07:47,112][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:07:47,441][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:07:47,770][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:07:48,100][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:07:48,431][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:07:48,762][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:07:49,096][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:07:49,430][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:07:49,753][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:07:50,080][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:07:50,408][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:07:50,736][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:07:51,067][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:07:51,393][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:07:52,163][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:07:52,893][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:07:52,895][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:07:52,896][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:07:53,834][__main__][INFO] - Iteration 333 took 23s (38.28% Gen, 57.68% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 17m 38s. Estimated total time: 19h 21m 43s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 37s.
[2025-11-13 10:07:53,836][__main__][INFO] - Starting iteration 333.
[2025-11-13 10:07:53,839][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:07:53,840][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:03,725][__main__][INFO] - Number of regex retries in iteration 333: 0
[2025-11-13 10:08:03,726][__main__][INFO] - agents played in iteration 333 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:08:04,178][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:04,217][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:04,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:04,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:04,296][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:04,296][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:05,021][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:08:05,318][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:08:05,645][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:08:05,973][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:08:06,300][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:08:06,626][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:08:06,953][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:08:07,279][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:08:07,605][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:08:07,932][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:08:08,258][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:08:08,585][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:08:08,912][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:08:09,240][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:08:09,566][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:08:09,894][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:08:10,221][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:08:10,547][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:08:10,874][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:08:11,201][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:08:11,530][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:08:11,857][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:08:12,188][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:08:12,518][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:08:12,846][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:08:13,174][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:08:13,515][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:08:13,843][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:08:14,171][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:08:14,499][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:08:14,831][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:08:15,158][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:08:15,486][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:08:16,251][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:08:16,956][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:08:16,958][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:08:16,960][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:08:17,910][__main__][INFO] - Iteration 334 took 24s (41.07% Gen, 54.98% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 59m 8s. Estimated total time: 20h 3m 37s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 7s, 500 more iterations: 3h 20m 36s.
[2025-11-13 10:08:17,912][__main__][INFO] - Starting iteration 334.
[2025-11-13 10:08:17,915][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:08:17,916][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:27,194][__main__][INFO] - Number of regex retries in iteration 334: 0
[2025-11-13 10:08:27,194][__main__][INFO] - agents played in iteration 334 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:08:27,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:27,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:27,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:27,762][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:27,763][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:27,763][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:28,525][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:08:28,822][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:08:29,151][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:08:29,479][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:08:29,807][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:08:30,134][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:08:30,461][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:08:30,793][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:08:31,122][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:08:31,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:08:31,776][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:08:32,109][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:08:32,443][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:08:32,776][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:08:33,103][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:08:33,434][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:08:33,761][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:08:34,088][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:08:34,415][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:08:34,744][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:08:35,071][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:08:35,399][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:08:35,736][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:08:36,061][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:08:36,389][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:08:36,718][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:08:37,049][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:08:37,376][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:08:37,704][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:08:38,031][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:08:38,359][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:08:38,689][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:08:39,017][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:08:39,777][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:08:40,489][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:08:40,491][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:08:40,493][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:08:41,525][__main__][INFO] - Iteration 335 took 23s (39.29% Gen, 56.32% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 35m 38s. Estimated total time: 19h 40m 31s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 21s, 500 more iterations: 3h 16m 45s.
[2025-11-13 10:08:41,527][__main__][INFO] - Starting iteration 335.
[2025-11-13 10:08:41,530][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:08:41,531][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:08:50,980][__main__][INFO] - Number of regex retries in iteration 335: 0
[2025-11-13 10:08:50,980][__main__][INFO] - agents played in iteration 335 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:08:51,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:51,455][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:51,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:51,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:08:51,534][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:08:51,535][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:08:52,311][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:08:52,608][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:08:52,942][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:08:53,266][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:08:53,592][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:08:53,919][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:08:54,250][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:08:54,578][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:08:54,912][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:08:55,241][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:08:55,570][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:08:55,897][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:08:56,230][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:08:56,564][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:08:56,892][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:08:57,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:08:57,560][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:08:57,886][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:08:58,223][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:08:58,551][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:08:58,879][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:08:59,207][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:08:59,543][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:08:59,868][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:09:00,199][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:09:00,527][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:09:00,856][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:09:01,184][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:09:01,512][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:09:01,841][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:09:02,168][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:09:02,501][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:09:02,829][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:09:03,587][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:09:04,288][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:04,290][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:04,292][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:09:05,395][__main__][INFO] - Iteration 336 took 23s (39.59% Gen, 55.78% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 48m 2s. Estimated total time: 19h 53m 19s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 46s, 500 more iterations: 3h 18m 53s.
[2025-11-13 10:09:05,397][__main__][INFO] - Starting iteration 336.
[2025-11-13 10:09:05,400][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:09:05,401][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:09:14,893][__main__][INFO] - Number of regex retries in iteration 336: 0
[2025-11-13 10:09:14,894][__main__][INFO] - agents played in iteration 336 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:09:15,341][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:15,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:15,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:15,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:15,462][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:09:15,462][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:09:16,226][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:09:16,523][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:09:16,853][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:09:17,186][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:09:17,515][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:09:17,844][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:09:18,176][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:09:18,506][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:09:18,836][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:09:19,165][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:09:19,496][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:09:19,825][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:09:20,155][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:09:20,481][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:09:20,807][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:09:21,141][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:09:21,467][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:09:21,795][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:09:22,124][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:09:22,455][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:09:22,783][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:09:23,110][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:09:23,438][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:09:23,766][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:09:24,093][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:09:24,424][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:09:24,754][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:09:25,081][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:09:25,408][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:09:25,741][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:09:26,066][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:09:26,393][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:09:26,721][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:09:27,476][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:09:28,185][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:28,186][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:28,187][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:09:29,120][__main__][INFO] - Iteration 337 took 23s (40.02% Gen, 56.04% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 40m 21s. Estimated total time: 19h 46m 1s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 32s, 500 more iterations: 3h 17m 40s.
[2025-11-13 10:09:29,122][__main__][INFO] - Starting iteration 337.
[2025-11-13 10:09:29,125][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:09:29,126][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:09:38,740][__main__][INFO] - Number of regex retries in iteration 337: 0
[2025-11-13 10:09:38,741][__main__][INFO] - agents played in iteration 337 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:09:39,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:39,226][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:39,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:39,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:09:39,307][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:09:39,308][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:09:40,073][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:09:40,372][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:09:40,699][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:09:41,028][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:09:41,357][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:09:41,684][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:09:42,010][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:09:42,336][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:09:42,666][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:09:42,997][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:09:43,324][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:09:43,650][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:09:43,977][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:09:44,310][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:09:44,638][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:09:44,966][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:09:45,293][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:09:45,620][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:09:45,951][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:09:46,279][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:09:46,609][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:09:46,942][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:09:47,265][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:09:47,593][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:09:47,923][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:09:48,252][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:09:48,581][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:09:48,911][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:09:49,239][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:09:49,567][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:09:49,903][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:09:50,231][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:09:50,559][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:09:51,322][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:09:52,039][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:09:52,041][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:09:52,043][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:09:53,030][__main__][INFO] - Iteration 338 took 23s (40.22% Gen, 55.65% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 49m 12s. Estimated total time: 19h 55m 16s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 12s.
[2025-11-13 10:09:53,032][__main__][INFO] - Starting iteration 338.
[2025-11-13 10:09:53,035][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:09:53,035][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:10:03,227][__main__][INFO] - Number of regex retries in iteration 338: 0
[2025-11-13 10:10:03,228][__main__][INFO] - agents played in iteration 338 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:10:03,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:03,724][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:03,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:03,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:03,804][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:10:03,804][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:10:04,592][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:10:04,888][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:10:05,214][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:10:05,540][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:10:05,868][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:10:06,197][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:10:06,526][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:10:06,853][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:10:07,180][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:10:07,507][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:10:07,834][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:10:08,161][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:10:08,488][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:10:08,815][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:10:09,142][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:10:09,469][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:10:09,796][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:10:10,124][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:10:10,452][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:10:10,781][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:10:11,109][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:10:11,437][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:10:11,763][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:10:12,094][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:10:12,428][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:10:12,756][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:10:13,084][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:10:13,412][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:10:13,740][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:10:14,069][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:10:14,397][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:10:14,726][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:10:15,055][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:10:15,827][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:10:16,556][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:10:16,557][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:10:16,559][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:10:17,817][__main__][INFO] - Iteration 339 took 24s (41.13% Gen, 53.79% Train). Generation: 10s, Training: 13s. Estimated remaining time: 18h 32m 40s. Estimated total time: 20h 39m 9s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 18s, 500 more iterations: 3h 26m 31s.
[2025-11-13 10:10:17,819][__main__][INFO] - Starting iteration 339.
[2025-11-13 10:10:17,822][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:10:17,822][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:10:26,844][__main__][INFO] - Number of regex retries in iteration 339: 0
[2025-11-13 10:10:26,845][__main__][INFO] - agents played in iteration 339 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:10:27,297][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:27,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:27,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:27,416][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:27,417][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:10:27,417][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:10:28,172][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:10:28,470][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:10:28,797][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:10:29,124][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:10:29,453][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:10:29,781][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:10:30,107][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:10:30,437][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:10:30,760][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:10:31,086][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:10:31,413][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:10:31,747][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:10:32,068][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:10:32,394][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:10:32,721][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:10:33,053][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:10:33,374][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:10:33,702][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:10:34,029][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:10:34,359][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:10:34,687][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:10:35,015][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:10:35,343][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:10:35,672][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:10:36,006][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:10:36,334][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:10:36,661][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:10:36,989][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:10:37,324][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:10:37,654][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:10:37,981][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:10:38,308][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:10:38,650][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:10:39,402][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:10:40,134][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:10:40,135][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:10:40,137][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:10:41,088][__main__][INFO] - Iteration 340 took 23s (38.78% Gen, 57.13% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 16m 29s. Estimated total time: 19h 23m 22s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 46s, 500 more iterations: 3h 13m 53s.
[2025-11-13 10:10:41,090][__main__][INFO] - Starting iteration 340.
[2025-11-13 10:10:41,094][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 33 and human policies 1.
[2025-11-13 10:10:41,094][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:10:50,565][__main__][INFO] - Number of regex retries in iteration 340: 0
[2025-11-13 10:10:50,566][__main__][INFO] - agents played in iteration 340 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:10:51,002][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:51,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:51,083][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:51,123][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:10:51,124][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:10:51,124][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:10:51,909][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:10:52,206][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:10:52,538][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:10:52,868][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:10:53,196][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:10:53,523][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:10:53,849][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:10:54,176][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:10:54,505][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:10:54,832][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:10:55,159][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:10:55,488][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:10:55,818][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:10:56,145][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:10:56,472][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:10:56,799][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:10:57,130][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:10:57,457][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:10:57,785][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:10:58,112][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:10:58,440][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:10:58,768][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:10:59,097][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:10:59,423][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:10:59,756][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:11:00,079][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:11:00,407][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:11:00,734][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:11:01,066][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:11:01,390][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:11:01,719][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:11:02,047][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:11:02,375][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:11:03,124][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:11:03,863][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:11:03,865][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:11:03,866][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:11:06,000][__main__][INFO] - Iteration 341 took 24s (38.02% Gen, 53.40% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 38m 5s. Estimated total time: 20h 45m 22s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 30s, 500 more iterations: 3h 27m 33s.
[2025-11-13 10:11:06,002][__main__][INFO] - Starting iteration 341.
[2025-11-13 10:11:06,005][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:11:06,006][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:11:15,582][__main__][INFO] - Number of regex retries in iteration 341: 0
[2025-11-13 10:11:15,583][__main__][INFO] - agents played in iteration 341 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:11:16,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:16,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:16,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:16,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:16,156][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:11:16,156][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:11:16,949][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:11:17,248][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:11:17,576][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:11:17,903][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:11:18,239][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:11:18,574][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:11:18,906][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:11:19,232][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:11:19,563][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:11:19,890][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:11:20,218][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:11:20,549][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:11:20,875][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:11:21,203][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:11:21,533][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:11:21,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:11:22,193][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:11:22,520][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:11:22,851][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:11:23,178][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:11:23,508][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:11:23,836][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:11:24,164][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:11:24,491][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:11:24,828][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:11:25,156][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:11:25,487][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:11:25,814][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:11:26,154][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:11:26,483][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:11:26,810][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:11:27,137][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:11:27,466][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:11:28,238][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:11:28,956][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:11:28,958][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:11:28,959][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:11:29,891][__main__][INFO] - Iteration 342 took 23s (40.09% Gen, 56.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 46m 39s. Estimated total time: 19h 54m 20s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 48s, 500 more iterations: 3h 19m 3s.
[2025-11-13 10:11:29,893][__main__][INFO] - Starting iteration 342.
[2025-11-13 10:11:29,896][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:11:29,897][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:11:38,892][__main__][INFO] - Number of regex retries in iteration 342: 0
[2025-11-13 10:11:38,893][__main__][INFO] - agents played in iteration 342 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:11:39,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:39,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:39,417][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:39,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:11:39,457][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:11:39,457][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:11:40,214][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:11:40,511][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:11:40,839][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:11:41,168][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:11:41,495][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:11:41,831][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:11:42,162][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:11:42,495][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:11:42,829][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:11:43,159][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:11:43,493][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:11:43,821][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:11:44,150][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:11:44,490][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:11:44,822][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:11:45,152][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:11:45,482][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:11:45,811][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:11:46,139][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:11:46,468][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:11:46,796][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:11:47,134][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:11:47,463][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:11:47,790][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:11:48,118][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:11:48,458][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:11:48,785][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:11:49,114][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:11:49,443][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:11:49,773][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:11:50,100][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:11:50,428][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:11:50,764][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:11:51,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:11:52,255][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:11:52,256][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:11:52,258][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:11:53,175][__main__][INFO] - Iteration 343 took 23s (38.64% Gen, 57.41% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 15m 56s. Estimated total time: 19h 24m 1s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 0s.
[2025-11-13 10:11:53,178][__main__][INFO] - Starting iteration 343.
[2025-11-13 10:11:53,182][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:11:53,182][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:12:02,645][__main__][INFO] - Number of regex retries in iteration 343: 0
[2025-11-13 10:12:02,646][__main__][INFO] - agents played in iteration 343 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:12:03,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:03,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:03,182][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:03,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:03,223][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:12:03,223][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:12:03,966][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:12:04,265][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:12:04,593][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:12:04,921][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:12:05,252][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:12:05,585][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:12:05,915][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:12:06,242][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:12:06,572][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:12:06,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:12:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:12:07,562][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:12:07,892][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:12:08,226][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:12:08,560][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:12:08,890][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:12:09,216][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:12:09,551][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:12:09,878][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:12:10,209][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:12:10,537][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:12:10,872][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:12:11,195][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:12:11,523][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:12:11,850][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:12:12,182][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:12:12,509][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:12:12,837][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:12:13,167][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:12:13,494][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:12:13,823][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:12:14,154][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:12:14,483][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:12:15,231][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:12:15,939][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:12:15,941][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:12:15,942][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:12:16,872][__main__][INFO] - Iteration 344 took 23s (39.94% Gen, 56.12% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 36m 6s. Estimated total time: 19h 44m 34s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 29s, 500 more iterations: 3h 17m 25s.
[2025-11-13 10:12:16,874][__main__][INFO] - Starting iteration 344.
[2025-11-13 10:12:16,877][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:12:16,878][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:12:26,760][__main__][INFO] - Number of regex retries in iteration 344: 0
[2025-11-13 10:12:26,760][__main__][INFO] - agents played in iteration 344 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:12:27,200][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:27,241][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:27,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:27,321][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:27,322][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:12:27,322][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:12:28,057][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:12:28,354][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:12:28,682][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:12:29,011][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:12:29,347][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:12:29,675][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:12:30,006][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:12:30,334][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:12:30,669][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:12:31,003][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:12:31,331][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:12:31,662][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:12:31,988][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:12:32,317][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:12:32,650][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:12:32,984][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:12:33,311][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:12:33,646][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:12:33,980][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:12:34,308][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:12:34,640][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:12:34,968][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:12:35,295][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:12:35,623][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:12:35,951][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:12:36,280][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:12:36,607][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:12:36,934][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:12:37,270][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:12:37,599][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:12:37,927][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:12:38,254][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:12:38,582][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:12:39,353][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:12:40,087][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:12:40,089][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:12:40,091][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:12:41,033][__main__][INFO] - Iteration 345 took 24s (40.91% Gen, 55.18% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 58m 59s. Estimated total time: 20h 7m 51s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 15s, 500 more iterations: 3h 21m 18s.
[2025-11-13 10:12:41,036][__main__][INFO] - Starting iteration 345.
[2025-11-13 10:12:41,039][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:12:41,040][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:12:50,648][__main__][INFO] - Number of regex retries in iteration 345: 0
[2025-11-13 10:12:50,648][__main__][INFO] - agents played in iteration 345 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:12:51,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:51,135][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:51,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:51,214][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:12:51,214][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:12:51,215][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:12:51,942][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:12:52,240][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:12:52,567][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:12:52,894][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:12:53,222][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:12:53,549][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:12:53,875][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:12:54,208][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:12:54,544][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:12:54,874][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:12:55,202][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:12:55,534][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:12:55,864][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:12:56,195][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:12:56,526][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:12:56,857][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:12:57,185][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:12:57,514][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:12:57,843][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:12:58,176][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:12:58,498][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:12:58,824][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:12:59,153][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:12:59,487][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:12:59,808][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:13:00,137][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:13:00,464][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:13:00,795][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:13:01,123][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:13:01,450][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:13:01,777][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:13:02,104][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:13:02,433][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:13:03,194][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:13:03,904][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:13:03,906][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:13:03,907][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:13:04,921][__main__][INFO] - Iteration 346 took 23s (40.23% Gen, 55.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 44m 53s. Estimated total time: 19h 54m 10s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 48s, 500 more iterations: 3h 19m 1s.
[2025-11-13 10:13:04,923][__main__][INFO] - Starting iteration 346.
[2025-11-13 10:13:04,926][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:13:04,927][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:13:14,035][__main__][INFO] - Number of regex retries in iteration 346: 0
[2025-11-13 10:13:14,035][__main__][INFO] - agents played in iteration 346 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:13:14,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:14,514][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:14,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:14,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:14,593][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:13:14,594][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:13:15,315][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:13:15,612][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:13:15,942][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:13:16,273][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:13:16,601][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:13:16,930][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:13:17,258][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:13:17,589][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:13:17,928][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:13:18,259][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:13:18,589][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:13:18,922][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:13:19,259][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:13:19,587][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:13:19,916][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:13:20,244][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:13:20,583][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:13:20,911][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:13:21,239][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:13:21,570][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:13:21,897][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:13:22,225][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:13:22,554][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:13:22,910][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:13:23,239][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:13:23,566][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:13:23,893][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:13:24,222][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:13:24,550][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:13:24,878][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:13:25,205][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:13:25,533][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:13:25,861][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:13:26,607][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:13:27,315][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:13:27,317][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:13:27,318][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:13:28,507][__main__][INFO] - Iteration 347 took 23s (38.62% Gen, 56.33% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 29m 26s. Estimated total time: 19h 39m 6s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 18s, 500 more iterations: 3h 16m 31s.
[2025-11-13 10:13:28,509][__main__][INFO] - Starting iteration 347.
[2025-11-13 10:13:28,512][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:13:28,513][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:13:37,459][__main__][INFO] - Number of regex retries in iteration 347: 0
[2025-11-13 10:13:37,459][__main__][INFO] - agents played in iteration 347 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:13:37,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:37,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:37,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:38,038][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:13:38,039][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:13:38,039][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:13:38,762][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:13:39,059][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:13:39,388][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:13:39,715][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:13:40,041][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:13:40,369][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:13:40,697][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:13:41,026][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:13:41,353][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:13:41,681][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:13:42,010][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:13:42,339][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:13:42,667][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:13:42,995][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:13:43,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:13:43,655][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:13:43,983][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:13:44,310][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:13:44,643][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:13:44,970][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:13:45,298][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:13:45,626][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:13:45,958][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:13:46,287][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:13:46,615][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:13:46,942][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:13:47,281][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:13:47,609][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:13:47,937][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:13:48,264][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:13:48,595][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:13:48,923][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:13:49,250][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:13:50,006][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:13:50,709][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:13:50,711][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:13:50,712][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:13:51,653][__main__][INFO] - Iteration 348 took 23s (38.66% Gen, 57.27% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 7m 2s. Estimated total time: 19h 17m 5s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 34s, 500 more iterations: 3h 12m 50s.
[2025-11-13 10:13:51,655][__main__][INFO] - Starting iteration 348.
[2025-11-13 10:13:51,659][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:13:51,659][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:14:00,889][__main__][INFO] - Number of regex retries in iteration 348: 0
[2025-11-13 10:14:00,890][__main__][INFO] - agents played in iteration 348 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:14:01,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:01,375][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:01,414][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:01,453][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:01,454][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:14:01,454][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:14:02,180][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:14:02,478][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:14:02,810][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:14:03,134][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:14:03,462][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:14:03,789][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:14:04,115][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:14:04,441][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:14:04,768][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:14:05,096][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:14:05,428][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:14:05,761][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:14:06,088][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:14:06,417][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:14:06,747][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:14:07,089][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:14:07,416][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:14:07,744][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:14:08,071][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:14:08,402][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:14:08,730][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:14:09,057][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:14:09,384][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:14:09,712][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:14:10,040][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:14:10,368][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:14:10,695][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:14:11,023][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:14:11,350][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:14:11,679][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:14:12,007][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:14:12,334][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:14:12,663][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:14:13,418][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:14:14,145][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:14:14,146][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:14:14,148][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:14:15,083][__main__][INFO] - Iteration 349 took 23s (39.40% Gen, 56.60% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 20m 51s. Estimated total time: 19h 31m 17s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 2s, 500 more iterations: 3h 15m 12s.
[2025-11-13 10:14:15,085][__main__][INFO] - Starting iteration 349.
[2025-11-13 10:14:15,088][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:14:15,089][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:14:23,791][__main__][INFO] - Number of regex retries in iteration 349: 0
[2025-11-13 10:14:23,791][__main__][INFO] - agents played in iteration 349 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:14:24,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:24,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:24,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:24,365][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:24,366][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:14:24,366][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:14:25,441][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:14:25,739][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:14:26,067][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:14:26,394][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:14:26,722][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:14:27,049][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:14:27,377][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:14:27,705][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:14:28,036][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:14:28,362][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:14:28,689][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:14:29,023][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:14:29,355][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:14:29,694][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:14:30,025][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:14:30,355][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:14:30,694][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:14:31,021][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:14:31,348][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:14:31,677][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:14:32,011][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:14:32,343][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:14:32,671][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:14:33,005][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:14:33,326][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:14:33,654][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:14:33,983][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:14:34,316][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:14:34,641][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:14:34,969][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:14:35,297][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:14:35,625][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:14:35,953][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:14:36,720][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:14:37,445][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:14:37,446][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:14:37,448][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:14:38,630][__main__][INFO] - Iteration 350 took 23s (36.96% Gen, 58.01% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 26m 18s. Estimated total time: 19h 37m 8s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 11s.
[2025-11-13 10:14:38,632][__main__][INFO] - Starting iteration 350.
[2025-11-13 10:14:38,635][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 34 and human policies 1.
[2025-11-13 10:14:38,636][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:14:47,873][__main__][INFO] - Number of regex retries in iteration 350: 0
[2025-11-13 10:14:47,874][__main__][INFO] - agents played in iteration 350 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:14:48,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:48,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:48,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:48,777][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:14:48,778][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:14:48,778][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:14:49,499][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:14:49,797][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:14:50,124][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:14:50,451][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:14:50,779][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:14:51,112][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:14:51,448][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:14:51,782][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:14:52,109][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:14:52,437][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:14:52,763][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:14:53,092][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:14:53,420][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:14:53,746][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:14:54,078][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:14:54,407][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:14:54,734][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:14:55,064][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:14:55,396][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:14:55,725][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:14:56,053][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:14:56,382][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:14:56,709][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:14:57,038][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:14:57,365][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:14:57,694][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:14:58,022][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:14:58,349][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:14:58,677][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:14:59,006][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:14:59,333][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:14:59,661][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:14:59,989][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:15:00,741][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:15:01,471][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:15:01,472][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:15:01,474][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:15:03,450][__main__][INFO] - Iteration 351 took 24s (37.23% Gen, 54.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 29m 33s. Estimated total time: 20h 40m 47s. Time estimates for 10 more iterations: 4m 8s, 100 more iterations: 41m 21s, 500 more iterations: 3h 26m 47s.
[2025-11-13 10:15:03,452][__main__][INFO] - Starting iteration 351.
[2025-11-13 10:15:03,455][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:15:03,456][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:15:11,677][__main__][INFO] - Number of regex retries in iteration 351: 0
[2025-11-13 10:15:11,678][__main__][INFO] - agents played in iteration 351 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:15:12,120][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:12,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:12,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:12,575][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:12,576][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:15:12,576][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:15:13,306][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:15:13,603][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:15:13,931][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:15:14,259][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:15:14,587][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:15:14,920][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:15:15,249][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:15:15,580][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:15:15,913][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:15:16,242][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:15:16,577][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:15:16,918][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:15:17,245][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:15:17,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:15:17,902][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:15:18,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:15:18,563][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:15:18,890][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:15:19,218][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:15:19,546][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:15:19,874][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:15:20,201][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:15:20,528][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:15:20,855][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:15:21,183][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:15:21,511][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:15:21,840][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:15:22,169][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:15:22,497][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:15:22,826][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:15:23,156][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:15:23,484][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:15:23,812][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:15:24,561][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:15:25,281][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:15:25,283][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:15:25,285][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:15:26,319][__main__][INFO] - Iteration 352 took 22s (35.96% Gen, 59.51% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 51m 36s. Estimated total time: 19h 3m 14s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 6s, 500 more iterations: 3h 10m 32s.
[2025-11-13 10:15:26,321][__main__][INFO] - Starting iteration 352.
[2025-11-13 10:15:26,324][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:15:26,325][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:15:35,709][__main__][INFO] - Number of regex retries in iteration 352: 0
[2025-11-13 10:15:35,710][__main__][INFO] - agents played in iteration 352 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:15:36,149][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:36,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:36,229][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:36,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:36,269][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:15:36,270][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:15:37,012][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:15:37,309][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:15:37,639][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:15:37,966][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:15:38,295][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:15:38,625][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:15:38,953][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:15:39,283][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:15:39,611][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:15:39,938][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:15:40,266][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:15:40,605][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:15:40,934][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:15:41,264][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:15:41,591][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:15:41,927][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:15:42,254][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:15:42,582][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:15:42,912][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:15:43,238][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:15:43,566][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:15:43,894][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:15:44,222][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:15:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:15:44,883][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:15:45,215][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:15:45,542][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:15:45,870][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:15:46,200][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:15:46,529][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:15:46,858][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:15:47,184][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:15:47,512][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:15:48,269][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:15:48,994][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:15:48,995][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:15:48,996][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:15:50,144][__main__][INFO] - Iteration 353 took 23s (39.40% Gen, 55.78% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 38m 59s. Estimated total time: 19h 51m 0s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 42s, 500 more iterations: 3h 18m 30s.
[2025-11-13 10:15:50,146][__main__][INFO] - Starting iteration 353.
[2025-11-13 10:15:50,149][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:15:50,150][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:15:59,158][__main__][INFO] - Number of regex retries in iteration 353: 0
[2025-11-13 10:15:59,158][__main__][INFO] - agents played in iteration 353 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:15:59,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:59,647][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:59,687][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:59,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:15:59,727][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:15:59,727][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:16:00,457][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:16:00,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:16:01,082][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:16:01,411][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:16:01,739][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:16:02,067][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:16:02,397][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:16:02,726][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:16:03,052][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:16:03,380][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:16:03,707][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:16:04,039][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:16:04,363][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:16:04,693][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:16:05,023][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:16:05,352][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:16:05,680][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:16:06,010][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:16:06,338][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:16:06,669][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:16:07,007][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:16:07,335][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:16:07,664][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:16:07,992][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:16:08,328][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:16:08,656][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:16:08,984][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:16:09,321][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:16:09,647][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:16:09,978][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:16:10,306][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:16:10,639][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:16:10,962][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:16:11,705][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:16:12,458][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:16:12,459][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:16:12,461][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:16:13,710][__main__][INFO] - Iteration 354 took 23s (38.23% Gen, 56.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 25m 40s. Estimated total time: 19h 38m 5s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 20s.
[2025-11-13 10:16:13,712][__main__][INFO] - Starting iteration 354.
[2025-11-13 10:16:13,715][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:16:13,716][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:16:22,706][__main__][INFO] - Number of regex retries in iteration 354: 0
[2025-11-13 10:16:22,707][__main__][INFO] - agents played in iteration 354 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:16:23,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:23,187][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:23,226][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:23,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:23,266][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:16:23,267][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:16:23,989][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:16:24,288][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:16:24,615][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:16:24,942][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:16:25,276][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:16:25,604][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:16:25,931][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:16:26,259][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:16:26,593][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:16:26,928][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:16:27,257][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:16:27,584][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:16:27,917][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:16:28,244][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:16:28,573][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:16:28,900][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:16:29,238][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:16:29,558][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:16:29,886][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:16:30,214][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:16:30,546][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:16:30,871][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:16:31,200][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:16:31,530][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:16:31,859][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:16:32,185][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:16:32,513][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:16:32,841][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:16:33,169][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:16:33,500][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:16:33,828][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:16:34,156][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:16:34,486][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:16:35,218][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:16:35,964][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:16:35,965][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:16:35,967][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:16:37,098][__main__][INFO] - Iteration 355 took 23s (38.45% Gen, 56.71% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 16m 21s. Estimated total time: 19h 29m 9s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 51s.
[2025-11-13 10:16:37,100][__main__][INFO] - Starting iteration 355.
[2025-11-13 10:16:37,103][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:16:37,104][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:16:46,348][__main__][INFO] - Number of regex retries in iteration 355: 0
[2025-11-13 10:16:46,348][__main__][INFO] - agents played in iteration 355 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:16:46,792][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:46,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:46,874][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:46,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:16:46,914][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:16:46,914][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:16:47,669][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:16:47,967][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:16:48,296][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:16:48,624][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:16:48,951][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:16:49,281][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:16:49,609][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:16:49,937][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:16:50,265][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:16:50,593][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:16:50,920][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:16:51,248][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:16:51,576][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:16:51,903][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:16:52,231][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:16:52,559][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:16:52,886][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:16:53,227][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:16:53,556][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:16:53,883][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:16:54,211][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:16:54,539][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:16:54,867][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:16:55,194][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:16:55,523][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:16:55,853][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:16:56,181][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:16:56,508][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:16:56,843][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:16:57,168][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:16:57,499][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:16:57,826][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:16:58,154][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:16:58,886][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:16:59,647][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:16:59,648][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:16:59,650][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:17:00,642][__main__][INFO] - Iteration 356 took 23s (39.27% Gen, 56.51% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 23m 46s. Estimated total time: 19h 36m 58s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 13s, 500 more iterations: 3h 16m 9s.
[2025-11-13 10:17:00,644][__main__][INFO] - Starting iteration 356.
[2025-11-13 10:17:00,648][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:17:00,649][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:17:08,441][mllm.models.large_language_model_local][WARNING] - Response <A><tool_call>
user
Last round, the other agent played <B>. did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 10:17:10,357][__main__][INFO] - Number of regex retries in iteration 356: 1
[2025-11-13 10:17:10,358][__main__][INFO] - agents played in iteration 356 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:17:10,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:10,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:10,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:10,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:10,916][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:17:10,917][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:17:11,692][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:17:11,987][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:17:12,312][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:17:12,643][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:17:12,962][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:17:13,286][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:17:13,611][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:17:13,945][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:17:14,269][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:17:14,595][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:17:14,921][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:17:15,255][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:17:15,573][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:17:15,899][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:17:16,224][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:17:16,556][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:17:16,876][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:17:17,203][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:17:17,529][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:17:17,861][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:17:18,178][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:17:18,504][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:17:18,829][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:17:19,161][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:17:19,480][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:17:19,805][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:17:20,129][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:17:20,461][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:17:20,781][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:17:21,107][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:17:21,435][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:17:21,767][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:17:22,088][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:17:22,813][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:17:23,577][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:17:23,579][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:17:23,580][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:17:24,592][__main__][INFO] - Iteration 357 took 23s (40.55% Gen, 55.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 43m 39s. Estimated total time: 19h 57m 15s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 54s, 500 more iterations: 3h 19m 32s.
[2025-11-13 10:17:24,594][__main__][INFO] - Starting iteration 357.
[2025-11-13 10:17:24,597][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:17:24,598][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:17:34,137][__main__][INFO] - Number of regex retries in iteration 357: 0
[2025-11-13 10:17:34,137][__main__][INFO] - agents played in iteration 357 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:17:34,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:34,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:34,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:34,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:34,702][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:17:34,703][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:17:35,447][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:17:35,830][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:17:36,168][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:17:36,495][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:17:36,820][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:17:37,145][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:17:37,469][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:17:37,798][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:17:38,124][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:17:38,449][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:17:38,775][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:17:39,100][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:17:39,426][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:17:39,751][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:17:40,076][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:17:40,402][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:17:40,727][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:17:41,053][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:17:41,378][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:17:41,702][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:17:42,029][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:17:42,354][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:17:42,680][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:17:43,005][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:17:43,330][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:17:43,655][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:17:43,982][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:17:44,310][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:17:44,635][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:17:44,962][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:17:45,290][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:17:45,616][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:17:45,942][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:17:46,782][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:17:47,574][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:17:47,575][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:17:47,577][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:17:48,647][__main__][INFO] - Iteration 358 took 24s (39.66% Gen, 55.88% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 48m 33s. Estimated total time: 20h 2m 33s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 5s, 500 more iterations: 3h 20m 25s.
[2025-11-13 10:17:48,650][__main__][INFO] - Starting iteration 358.
[2025-11-13 10:17:48,653][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:17:48,653][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:17:57,982][__main__][INFO] - Number of regex retries in iteration 358: 0
[2025-11-13 10:17:57,983][__main__][INFO] - agents played in iteration 358 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:17:58,437][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:58,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:58,504][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:58,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:17:58,540][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:17:58,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:17:59,329][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:17:59,625][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:17:59,951][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:18:00,277][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:18:00,610][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:18:00,937][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:18:01,263][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:18:01,590][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:18:01,915][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:18:02,239][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:18:02,564][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:18:02,889][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:18:03,213][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:18:03,539][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:18:03,864][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:18:04,189][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:18:04,514][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:18:04,839][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:18:05,163][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:18:05,489][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:18:05,813][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:18:06,138][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:18:06,464][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:18:06,789][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:18:07,114][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:18:07,440][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:18:07,765][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:18:08,092][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:18:08,423][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:18:08,748][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:18:09,077][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:18:09,403][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:18:09,728][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:18:10,462][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:18:11,198][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:18:11,199][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:18:11,201][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:18:12,202][__main__][INFO] - Iteration 359 took 23s (39.61% Gen, 56.13% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 23m 5s. Estimated total time: 19h 37m 29s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 14s.
[2025-11-13 10:18:12,204][__main__][INFO] - Starting iteration 359.
[2025-11-13 10:18:12,207][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:18:12,208][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:18:21,277][__main__][INFO] - Number of regex retries in iteration 359: 0
[2025-11-13 10:18:21,278][__main__][INFO] - agents played in iteration 359 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:18:21,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:21,755][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:21,788][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:21,821][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:21,821][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:18:21,822][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:18:22,528][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:18:22,823][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:18:23,148][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:18:23,478][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:18:23,797][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:18:24,124][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:18:24,449][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:18:24,776][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:18:25,102][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:18:25,428][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:18:25,754][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:18:26,079][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:18:26,404][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:18:26,730][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:18:27,056][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:18:27,381][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:18:27,705][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:18:28,031][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:18:28,356][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:18:28,680][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:18:29,006][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:18:29,331][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:18:29,657][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:18:29,983][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:18:30,308][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:18:30,634][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:18:30,959][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:18:31,284][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:18:31,609][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:18:31,938][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:18:32,263][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:18:32,592][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:18:32,919][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:18:33,664][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:18:34,396][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:18:34,397][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:18:34,399][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:18:35,385][__main__][INFO] - Iteration 360 took 23s (39.13% Gen, 56.61% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 4m 11s. Estimated total time: 19h 18m 57s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 9s.
[2025-11-13 10:18:35,388][__main__][INFO] - Starting iteration 360.
[2025-11-13 10:18:35,391][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 35 and human policies 1.
[2025-11-13 10:18:35,392][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:18:44,653][__main__][INFO] - Number of regex retries in iteration 360: 0
[2025-11-13 10:18:44,654][__main__][INFO] - agents played in iteration 360 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:18:45,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:45,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:45,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:45,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:18:45,214][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:18:45,214][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:18:45,944][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:18:46,239][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:18:46,562][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:18:46,887][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:18:47,212][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:18:47,537][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:18:47,861][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:18:48,186][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:18:48,512][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:18:48,836][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:18:49,161][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:18:49,486][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:18:49,812][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:18:50,136][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:18:50,461][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:18:50,786][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:18:51,110][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:18:51,435][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:18:51,760][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:18:52,085][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:18:52,410][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:18:52,734][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:18:53,059][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:18:53,384][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:18:53,710][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:18:54,035][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:18:54,361][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:18:54,686][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:18:55,011][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:18:55,345][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:18:55,675][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:18:56,000][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:18:56,326][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:18:57,065][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:18:57,828][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:18:57,830][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:18:57,832][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:18:59,782][__main__][INFO] - Iteration 361 took 24s (37.97% Gen, 54.03% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 4m 24s. Estimated total time: 20h 19m 35s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 39s, 500 more iterations: 3h 23m 15s.
[2025-11-13 10:18:59,784][__main__][INFO] - Starting iteration 361.
[2025-11-13 10:18:59,788][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:18:59,788][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:19:09,000][__main__][INFO] - Number of regex retries in iteration 361: 0
[2025-11-13 10:19:09,001][__main__][INFO] - agents played in iteration 361 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:19:09,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:09,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:09,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:09,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:09,574][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:19:09,574][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:19:10,282][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:19:10,576][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:19:10,902][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:19:11,229][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:19:11,555][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:19:11,882][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:19:12,210][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:19:12,535][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:19:12,862][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:19:13,187][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:19:13,514][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:19:13,839][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:19:14,164][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:19:14,490][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:19:14,816][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:19:15,141][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:19:15,467][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:19:15,793][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:19:16,118][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:19:16,442][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:19:16,767][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:19:17,093][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:19:17,418][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:19:17,742][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:19:18,066][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:19:18,392][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:19:18,716][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:19:19,049][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:19:19,374][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:19:19,700][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:19:20,025][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:19:20,349][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:19:20,676][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:19:21,442][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:19:22,184][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:19:22,246][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:19:22,248][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:19:23,392][__main__][INFO] - Iteration 362 took 23s (39.03% Gen, 56.12% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 24m 38s. Estimated total time: 19h 40m 13s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 20s, 500 more iterations: 3h 16m 42s.
[2025-11-13 10:19:23,394][__main__][INFO] - Starting iteration 362.
[2025-11-13 10:19:23,397][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:19:23,397][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:19:33,055][__main__][INFO] - Number of regex retries in iteration 362: 0
[2025-11-13 10:19:33,056][__main__][INFO] - agents played in iteration 362 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:19:33,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:33,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:33,578][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:33,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:33,613][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:19:33,613][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:19:34,363][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:19:34,660][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:19:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:19:35,313][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:19:35,638][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:19:35,964][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:19:36,292][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:19:36,618][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:19:36,944][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:19:37,270][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:19:37,595][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:19:37,921][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:19:38,246][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:19:38,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:19:38,899][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:19:39,224][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:19:39,549][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:19:39,875][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:19:40,200][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:19:40,526][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:19:40,851][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:19:41,177][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:19:41,502][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:19:41,827][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:19:42,153][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:19:42,478][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:19:42,803][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:19:43,127][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:19:43,452][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:19:43,779][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:19:44,104][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:19:44,429][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:19:44,759][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:19:45,500][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:19:46,241][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:19:46,242][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:19:46,244][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:19:47,435][__main__][INFO] - Iteration 363 took 24s (40.18% Gen, 54.86% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 45m 56s. Estimated total time: 20h 1m 55s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 19s.
[2025-11-13 10:19:47,437][__main__][INFO] - Starting iteration 363.
[2025-11-13 10:19:47,440][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:19:47,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:19:56,986][__main__][INFO] - Number of regex retries in iteration 363: 0
[2025-11-13 10:19:56,986][__main__][INFO] - agents played in iteration 363 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:19:57,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:57,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:57,501][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:57,533][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:19:57,534][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:19:57,534][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:19:58,266][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:19:58,560][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:19:58,883][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:19:59,207][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:19:59,533][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:19:59,863][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:20:00,189][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:20:00,513][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:20:00,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:20:01,164][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:20:01,494][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:20:01,821][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:20:02,149][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:20:02,477][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:20:02,801][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:20:03,127][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:20:03,451][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:20:03,777][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:20:04,102][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:20:04,427][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:20:04,753][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:20:05,079][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:20:05,405][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:20:05,731][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:20:06,056][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:20:06,380][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:20:06,706][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:20:07,032][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:20:07,357][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:20:07,682][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:20:08,006][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:20:08,332][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:20:08,657][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:20:09,390][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:20:10,138][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:20:10,140][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:20:10,141][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:20:11,115][__main__][INFO] - Iteration 364 took 23s (40.32% Gen, 55.56% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 27m 25s. Estimated total time: 19h 43m 47s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 27s, 500 more iterations: 3h 17m 17s.
[2025-11-13 10:20:11,117][__main__][INFO] - Starting iteration 364.
[2025-11-13 10:20:11,120][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:20:11,121][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:20:20,346][__main__][INFO] - Number of regex retries in iteration 364: 0
[2025-11-13 10:20:20,347][__main__][INFO] - agents played in iteration 364 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:20:20,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:20,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:20,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:20,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:20,900][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:20:20,900][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:20:21,622][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:20:21,918][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:20:22,242][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:20:22,565][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:20:22,890][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:20:23,214][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:20:23,542][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:20:23,868][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:20:24,193][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:20:24,518][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:20:24,844][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:20:25,168][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:20:25,494][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:20:25,819][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:20:26,147][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:20:26,474][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:20:26,801][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:20:27,127][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:20:27,453][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:20:27,778][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:20:28,103][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:20:28,429][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:20:28,754][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:20:29,080][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:20:29,405][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:20:29,731][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:20:30,056][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:20:30,382][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:20:30,707][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:20:31,032][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:20:31,359][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:20:31,683][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:20:32,009][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:20:32,749][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:20:33,493][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:20:33,499][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:20:33,500][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:20:34,478][__main__][INFO] - Iteration 365 took 23s (39.50% Gen, 56.31% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 11m 11s. Estimated total time: 19h 27m 56s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 55s, 500 more iterations: 3h 14m 39s.
[2025-11-13 10:20:34,480][__main__][INFO] - Starting iteration 365.
[2025-11-13 10:20:34,484][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:20:34,484][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:20:43,086][__main__][INFO] - Number of regex retries in iteration 365: 0
[2025-11-13 10:20:43,087][__main__][INFO] - agents played in iteration 365 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:20:43,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:43,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:43,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:43,630][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:20:43,631][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:20:43,631][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:20:44,373][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:20:44,668][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:20:44,993][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:20:45,319][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:20:45,652][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:20:45,983][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:20:46,311][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:20:46,637][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:20:46,966][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:20:47,294][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:20:47,624][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:20:47,949][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:20:48,288][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:20:48,612][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:20:48,939][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:20:49,265][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:20:49,597][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:20:49,923][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:20:50,251][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:20:50,576][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:20:50,903][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:20:51,228][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:20:51,555][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:20:51,880][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:20:52,206][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:20:52,531][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:20:52,857][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:20:53,181][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:20:53,507][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:20:53,833][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:20:54,158][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:20:54,488][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:20:54,810][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:20:55,551][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:20:56,313][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:20:56,315][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:20:56,316][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:20:57,318][__main__][INFO] - Iteration 366 took 22s (37.67% Gen, 57.93% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 44m 36s. Estimated total time: 19h 1m 44s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 3s, 500 more iterations: 3h 10m 17s.
[2025-11-13 10:20:57,320][__main__][INFO] - Starting iteration 366.
[2025-11-13 10:20:57,323][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:20:57,324][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:21:06,293][__main__][INFO] - Number of regex retries in iteration 366: 0
[2025-11-13 10:21:06,294][__main__][INFO] - agents played in iteration 366 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:21:06,766][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:06,800][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:06,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:06,865][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:06,865][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:21:06,866][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:21:07,598][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:21:07,900][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:21:08,221][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:21:08,545][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:21:08,871][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:21:09,205][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:21:09,525][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:21:09,855][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:21:10,181][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:21:10,515][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:21:10,835][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:21:11,162][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:21:11,489][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:21:11,818][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:21:12,141][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:21:12,467][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:21:12,793][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:21:13,122][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:21:13,447][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:21:13,773][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:21:14,097][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:21:14,422][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:21:14,748][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:21:15,075][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:21:15,401][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:21:15,727][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:21:16,052][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:21:16,378][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:21:16,704][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:21:17,029][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:21:17,354][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:21:17,680][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:21:18,006][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:21:18,755][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:21:19,493][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:21:19,495][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:21:19,496][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:21:20,477][__main__][INFO] - Iteration 367 took 23s (38.74% Gen, 57.02% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 0m 13s. Estimated total time: 19h 17m 44s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 35s, 500 more iterations: 3h 12m 57s.
[2025-11-13 10:21:20,480][__main__][INFO] - Starting iteration 367.
[2025-11-13 10:21:20,483][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:21:20,484][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:21:29,549][__main__][INFO] - Number of regex retries in iteration 367: 0
[2025-11-13 10:21:29,550][__main__][INFO] - agents played in iteration 367 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:21:30,014][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:30,051][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:30,087][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:30,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:30,134][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:21:30,135][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:21:30,880][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:21:31,174][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:21:31,500][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:21:31,825][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:21:32,149][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:21:32,473][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:21:32,800][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:21:33,126][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:21:33,455][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:21:33,783][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:21:34,112][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:21:34,437][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:21:34,763][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:21:35,090][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:21:35,416][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:21:35,741][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:21:36,066][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:21:36,390][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:21:36,717][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:21:37,042][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:21:37,367][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:21:37,691][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:21:38,016][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:21:38,341][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:21:38,666][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:21:38,991][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:21:39,318][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:21:39,642][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:21:39,968][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:21:40,293][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:21:40,618][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:21:40,943][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:21:41,268][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:21:42,028][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:21:42,739][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:21:42,740][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:21:42,742][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:21:43,934][__main__][INFO] - Iteration 368 took 23s (38.66% Gen, 56.25% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 14m 39s. Estimated total time: 19h 32m 34s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 5s, 500 more iterations: 3h 15m 25s.
[2025-11-13 10:21:43,936][__main__][INFO] - Starting iteration 368.
[2025-11-13 10:21:43,939][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:21:43,939][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:21:52,865][__main__][INFO] - Number of regex retries in iteration 368: 0
[2025-11-13 10:21:52,865][__main__][INFO] - agents played in iteration 368 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:21:53,315][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:53,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:53,383][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:53,417][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:21:53,418][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:21:53,418][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:21:54,170][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:21:54,465][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:21:54,789][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:21:55,115][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:21:55,441][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:21:55,771][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:21:56,100][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:21:56,421][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:21:56,746][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:21:57,072][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:21:57,400][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:21:57,724][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:21:58,056][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:21:58,382][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:21:58,714][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:21:59,044][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:21:59,375][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:21:59,702][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:22:00,033][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:22:00,369][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:22:00,697][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:22:01,023][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:22:01,350][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:22:01,681][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:22:02,005][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:22:02,331][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:22:02,656][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:22:02,982][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:22:03,309][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:22:03,635][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:22:03,960][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:22:04,285][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:22:04,610][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:22:05,365][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:22:06,071][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:22:06,072][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:22:06,074][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:22:07,126][__main__][INFO] - Iteration 369 took 23s (38.49% Gen, 56.96% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 1m 6s. Estimated total time: 19h 19m 25s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 14s.
[2025-11-13 10:22:07,129][__main__][INFO] - Starting iteration 369.
[2025-11-13 10:22:07,132][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:22:07,133][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:22:15,812][__main__][INFO] - Number of regex retries in iteration 369: 0
[2025-11-13 10:22:15,813][__main__][INFO] - agents played in iteration 369 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:22:16,266][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:16,642][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:16,674][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:16,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:16,708][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:22:16,708][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:22:17,440][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:22:17,736][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:22:18,060][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:22:18,384][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:22:18,712][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:22:19,037][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:22:19,367][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:22:19,692][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:22:20,017][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:22:20,342][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:22:20,667][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:22:20,992][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:22:21,318][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:22:21,645][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:22:21,975][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:22:22,299][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:22:22,624][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:22:22,949][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:22:23,274][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:22:23,601][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:22:23,929][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:22:24,256][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:22:24,582][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:22:24,908][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:22:25,238][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:22:25,563][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:22:25,888][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:22:26,212][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:22:26,538][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:22:26,863][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:22:27,189][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:22:27,514][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:22:27,840][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:22:28,593][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:22:29,294][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:22:29,301][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:22:29,303][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:22:30,307][__main__][INFO] - Iteration 370 took 23s (37.45% Gen, 58.21% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 0m 5s. Estimated total time: 19h 18m 47s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 7s.
[2025-11-13 10:22:30,309][__main__][INFO] - Starting iteration 370.
[2025-11-13 10:22:30,312][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 36 and human policies 1.
[2025-11-13 10:22:30,312][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:22:39,734][__main__][INFO] - Number of regex retries in iteration 370: 0
[2025-11-13 10:22:39,734][__main__][INFO] - agents played in iteration 370 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:22:40,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:40,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:40,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:40,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:22:40,289][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:22:40,289][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:22:41,015][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:22:41,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:22:41,635][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:22:41,959][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:22:42,287][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:22:42,614][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:22:42,940][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:22:43,265][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:22:43,596][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:22:43,921][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:22:44,246][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:22:44,571][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:22:44,904][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:22:45,229][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:22:45,556][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:22:45,879][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:22:46,216][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:22:46,545][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:22:46,870][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:22:47,196][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:22:47,524][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:22:47,851][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:22:48,178][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:22:48,505][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:22:48,829][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:22:49,155][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:22:49,480][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:22:49,805][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:22:50,131][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:22:50,458][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:22:50,783][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:22:51,108][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:22:51,433][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:22:52,203][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:22:52,927][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:22:52,928][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:22:52,930][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:22:54,912][__main__][INFO] - Iteration 371 took 24s (38.30% Gen, 53.64% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 10m 55s. Estimated total time: 20h 30m 1s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 0s, 500 more iterations: 3h 25m 0s.
[2025-11-13 10:22:54,914][__main__][INFO] - Starting iteration 371.
[2025-11-13 10:22:54,917][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:22:54,917][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:23:03,798][__main__][INFO] - Number of regex retries in iteration 371: 0
[2025-11-13 10:23:03,799][__main__][INFO] - agents played in iteration 371 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:23:04,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:04,287][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:04,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:04,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:04,353][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:23:04,353][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:23:05,080][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:23:05,373][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:23:05,698][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:23:06,027][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:23:06,351][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:23:06,676][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:23:07,004][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:23:07,331][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:23:07,660][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:23:07,989][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:23:08,322][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:23:08,642][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:23:08,969][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:23:09,295][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:23:09,627][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:23:09,948][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:23:10,274][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:23:10,599][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:23:10,930][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:23:11,250][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:23:11,575][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:23:11,902][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:23:12,234][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:23:12,552][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:23:12,877][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:23:13,202][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:23:13,526][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:23:13,852][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:23:14,177][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:23:14,502][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:23:14,827][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:23:15,152][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:23:15,477][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:23:16,223][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:23:16,954][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:23:16,956][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:23:16,958][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:23:17,915][__main__][INFO] - Iteration 372 took 22s (38.61% Gen, 57.22% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 50m 29s. Estimated total time: 19h 9m 58s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 19s, 500 more iterations: 3h 11m 39s.
[2025-11-13 10:23:17,918][__main__][INFO] - Starting iteration 372.
[2025-11-13 10:23:17,921][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:23:17,922][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:23:27,216][__main__][INFO] - Number of regex retries in iteration 372: 0
[2025-11-13 10:23:27,217][__main__][INFO] - agents played in iteration 372 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:23:27,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:27,696][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:27,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:27,762][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:27,762][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:23:27,763][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:23:28,499][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:23:28,796][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:23:29,121][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:23:29,451][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:23:29,775][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:23:30,100][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:23:30,428][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:23:30,754][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:23:31,081][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:23:31,409][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:23:31,734][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:23:32,060][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:23:32,390][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:23:32,715][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:23:33,040][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:23:33,369][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:23:33,699][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:23:34,025][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:23:34,349][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:23:34,677][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:23:35,002][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:23:35,328][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:23:35,652][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:23:35,978][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:23:36,312][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:23:36,638][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:23:36,963][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:23:37,287][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:23:37,612][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:23:37,936][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:23:38,262][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:23:38,587][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:23:38,912][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:23:39,669][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:23:40,370][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:23:40,371][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:23:40,373][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:23:41,584][__main__][INFO] - Iteration 373 took 23s (39.28% Gen, 55.60% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 23m 19s. Estimated total time: 19h 43m 11s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 26s, 500 more iterations: 3h 17m 11s.
[2025-11-13 10:23:41,586][__main__][INFO] - Starting iteration 373.
[2025-11-13 10:23:41,589][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:23:41,589][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:23:50,053][__main__][INFO] - Number of regex retries in iteration 373: 0
[2025-11-13 10:23:50,053][__main__][INFO] - agents played in iteration 373 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:23:50,524][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:50,558][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:50,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:50,624][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:23:50,625][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:23:50,625][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:23:51,698][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:23:51,999][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:23:52,323][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:23:52,649][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:23:52,977][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:23:53,308][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:23:53,635][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:23:53,964][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:23:54,293][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:23:54,628][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:23:54,956][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:23:55,283][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:23:55,608][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:23:55,937][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:23:56,264][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:23:56,589][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:23:56,915][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:23:57,241][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:23:57,568][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:23:57,894][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:23:58,221][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:23:58,546][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:23:58,872][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:23:59,197][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:23:59,523][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:23:59,851][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:24:00,176][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:24:00,500][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:24:00,824][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:24:01,149][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:24:01,475][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:24:01,800][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:24:02,126][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:24:02,970][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:24:03,699][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:24:03,701][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:24:03,702][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:24:04,694][__main__][INFO] - Iteration 374 took 23s (36.63% Gen, 59.07% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 55m 4s. Estimated total time: 19h 15m 20s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 33s.
[2025-11-13 10:24:04,697][__main__][INFO] - Starting iteration 374.
[2025-11-13 10:24:04,700][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:24:04,700][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:24:12,473][__main__][INFO] - Number of regex retries in iteration 374: 0
[2025-11-13 10:24:12,474][__main__][INFO] - agents played in iteration 374 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:24:12,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:13,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:13,362][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:13,395][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:13,396][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:24:13,396][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:24:14,143][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:24:14,440][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:24:14,766][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:24:15,091][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:24:15,419][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:24:15,748][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:24:16,074][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:24:16,399][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:24:16,723][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:24:17,049][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:24:17,376][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:24:17,701][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:24:18,025][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:24:18,349][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:24:18,674][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:24:19,000][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:24:19,329][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:24:19,650][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:24:19,977][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:24:20,304][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:24:20,635][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:24:20,956][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:24:21,287][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:24:21,618][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:24:21,942][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:24:22,267][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:24:22,593][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:24:22,919][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:24:23,244][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:24:23,569][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:24:23,893][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:24:24,218][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:24:24,544][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:24:25,310][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:24:26,019][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:24:26,020][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:24:26,022][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:24:26,947][__main__][INFO] - Iteration 375 took 22s (34.94% Gen, 60.90% Train). Generation: 7s, Training: 13s. Estimated remaining time: 16h 11m 47s. Estimated total time: 18h 32m 25s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 4s, 500 more iterations: 3h 5m 24s.
[2025-11-13 10:24:26,949][__main__][INFO] - Starting iteration 375.
[2025-11-13 10:24:26,952][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:24:26,952][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:24:35,999][__main__][INFO] - Number of regex retries in iteration 375: 0
[2025-11-13 10:24:36,000][__main__][INFO] - agents played in iteration 375 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:24:36,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:36,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:36,520][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:36,554][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:24:36,554][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:24:36,554][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:24:37,300][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:24:37,597][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:24:37,926][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:24:38,252][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:24:38,581][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:24:38,905][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:24:39,233][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:24:39,560][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:24:39,885][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:24:40,211][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:24:40,535][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:24:40,860][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:24:41,184][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:24:41,508][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:24:41,835][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:24:42,163][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:24:42,487][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:24:42,813][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:24:43,141][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:24:43,465][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:24:43,791][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:24:44,117][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:24:44,452][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:24:44,774][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:24:45,101][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:24:45,427][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:24:45,762][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:24:46,081][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:24:46,407][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:24:46,732][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:24:47,065][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:24:47,384][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:24:47,709][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:24:48,468][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:24:49,164][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:24:49,166][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:24:49,167][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:24:51,234][__main__][INFO] - Iteration 376 took 24s (37.26% Gen, 54.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 53m 7s. Estimated total time: 20h 14m 10s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 28s, 500 more iterations: 3h 22m 21s.
[2025-11-13 10:24:51,237][__main__][INFO] - Starting iteration 376.
[2025-11-13 10:24:51,240][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:24:51,240][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:25:00,295][__main__][INFO] - Number of regex retries in iteration 376: 0
[2025-11-13 10:25:00,296][__main__][INFO] - agents played in iteration 376 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:25:00,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:00,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:00,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:00,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:00,850][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:25:00,851][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:25:01,590][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:25:01,885][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:25:02,210][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:25:02,534][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:25:02,859][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:25:03,184][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:25:03,512][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:25:03,838][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:25:04,165][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:25:04,491][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:25:04,815][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:25:05,141][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:25:05,469][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:25:05,794][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:25:06,121][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:25:06,451][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:25:06,777][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:25:07,103][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:25:07,428][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:25:07,757][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:25:08,085][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:25:08,412][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:25:08,740][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:25:09,067][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:25:09,392][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:25:09,717][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:25:10,043][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:25:10,369][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:25:10,694][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:25:11,020][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:25:11,345][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:25:11,671][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:25:11,996][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:25:12,747][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:25:13,466][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:25:13,468][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:25:13,469][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:25:14,394][__main__][INFO] - Iteration 377 took 23s (39.11% Gen, 56.89% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 19s. Estimated total time: 19h 17m 45s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 35s, 500 more iterations: 3h 12m 57s.
[2025-11-13 10:25:14,396][__main__][INFO] - Starting iteration 377.
[2025-11-13 10:25:14,399][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:25:14,399][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:25:23,279][__main__][INFO] - Number of regex retries in iteration 377: 0
[2025-11-13 10:25:23,280][__main__][INFO] - agents played in iteration 377 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:25:23,731][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:23,764][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:23,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:23,830][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:23,831][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:25:23,831][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:25:24,561][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:25:24,857][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:25:25,186][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:25:25,511][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:25:25,837][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:25:26,162][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:25:26,488][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:25:26,812][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:25:27,136][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:25:27,463][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:25:27,787][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:25:28,111][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:25:28,436][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:25:28,760][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:25:29,084][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:25:29,408][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:25:29,736][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:25:30,062][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:25:30,388][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:25:30,714][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:25:31,040][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:25:31,365][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:25:31,695][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:25:32,024][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:25:32,350][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:25:32,675][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:25:33,001][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:25:33,327][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:25:33,653][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:25:33,979][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:25:34,304][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:25:34,629][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:25:34,954][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:25:35,701][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:25:36,396][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:25:36,397][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:25:36,399][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:25:37,354][__main__][INFO] - Iteration 378 took 22s (38.68% Gen, 57.15% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 46m 1s. Estimated total time: 19h 7m 50s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 18s.
[2025-11-13 10:25:37,356][__main__][INFO] - Starting iteration 378.
[2025-11-13 10:25:37,359][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:25:37,360][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:25:46,485][__main__][INFO] - Number of regex retries in iteration 378: 0
[2025-11-13 10:25:46,486][__main__][INFO] - agents played in iteration 378 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:25:46,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:46,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:47,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:47,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:25:47,041][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:25:47,041][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:25:47,779][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:25:48,074][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:25:48,400][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:25:48,724][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:25:49,048][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:25:49,375][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:25:49,700][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:25:50,025][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:25:50,355][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:25:50,682][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:25:51,008][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:25:51,334][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:25:51,660][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:25:51,990][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:25:52,321][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:25:52,651][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:25:52,978][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:25:53,305][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:25:53,633][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:25:53,964][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:25:54,290][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:25:54,620][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:25:54,947][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:25:55,273][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:25:55,598][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:25:55,923][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:25:56,248][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:25:56,574][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:25:56,900][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:25:57,225][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:25:57,551][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:25:57,880][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:25:58,205][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:25:58,953][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:25:59,685][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:25:59,686][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:25:59,688][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:26:00,663][__main__][INFO] - Iteration 379 took 23s (39.16% Gen, 56.65% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 3m 2s. Estimated total time: 19h 25m 14s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 50s, 500 more iterations: 3h 14m 12s.
[2025-11-13 10:26:00,666][__main__][INFO] - Starting iteration 379.
[2025-11-13 10:26:00,669][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:26:00,670][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:26:09,628][__main__][INFO] - Number of regex retries in iteration 379: 0
[2025-11-13 10:26:09,628][__main__][INFO] - agents played in iteration 379 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:26:10,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:10,115][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:10,148][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:10,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:10,182][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:26:10,182][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:26:10,930][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:26:11,224][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:26:11,550][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:26:11,875][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:26:12,199][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:26:12,523][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:26:12,848][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:26:13,172][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:26:13,496][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:26:13,822][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:26:14,154][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:26:14,480][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:26:14,805][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:26:15,130][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:26:15,459][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:26:15,785][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:26:16,110][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:26:16,436][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:26:16,764][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:26:17,089][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:26:17,414][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:26:17,739][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:26:18,065][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:26:18,390][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:26:18,720][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:26:19,046][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:26:19,371][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:26:19,696][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:26:20,022][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:26:20,348][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:26:20,674][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:26:21,000][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:26:21,325][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:26:22,102][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:26:22,819][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:26:22,822][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:26:22,823][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:26:23,722][__main__][INFO] - Iteration 380 took 23s (38.85% Gen, 57.23% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 50m 8s. Estimated total time: 19h 12m 43s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 25s, 500 more iterations: 3h 12m 7s.
[2025-11-13 10:26:23,724][__main__][INFO] - Starting iteration 380.
[2025-11-13 10:26:23,727][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 37 and human policies 1.
[2025-11-13 10:26:23,728][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:26:32,627][__main__][INFO] - Number of regex retries in iteration 380: 0
[2025-11-13 10:26:32,628][__main__][INFO] - agents played in iteration 380 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:26:33,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:33,113][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:33,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:33,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:33,180][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:26:33,181][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:26:34,302][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:26:34,597][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:26:34,925][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:26:35,251][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:26:35,575][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:26:35,901][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:26:36,230][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:26:36,555][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:26:36,886][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:26:37,213][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:26:37,542][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:26:37,866][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:26:38,193][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:26:38,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:26:38,856][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:26:39,183][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:26:39,510][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:26:39,839][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:26:40,167][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:26:40,497][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:26:40,820][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:26:41,146][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:26:41,473][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:26:41,805][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:26:42,129][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:26:42,459][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:26:42,778][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:26:43,103][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:26:43,431][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:26:43,764][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:26:44,082][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:26:44,407][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:26:44,734][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:26:45,503][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:26:46,199][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:26:46,201][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:26:46,202][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:26:48,211][__main__][INFO] - Iteration 381 took 24s (36.35% Gen, 55.44% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 1m 14s. Estimated total time: 20h 24m 13s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 48s, 500 more iterations: 3h 24m 2s.
[2025-11-13 10:26:48,213][__main__][INFO] - Starting iteration 381.
[2025-11-13 10:26:48,216][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:26:48,217][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:26:57,912][__main__][INFO] - Number of regex retries in iteration 381: 0
[2025-11-13 10:26:57,913][__main__][INFO] - agents played in iteration 381 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:26:58,372][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:58,406][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:58,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:58,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:26:58,474][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:26:58,474][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:26:59,218][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:26:59,513][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:26:59,848][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:27:00,172][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:27:00,499][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:27:00,825][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:27:01,150][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:27:01,477][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:27:01,805][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:27:02,130][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:27:02,464][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:27:02,794][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:27:03,119][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:27:03,448][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:27:03,775][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:27:04,101][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:27:04,427][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:27:04,758][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:27:05,079][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:27:05,405][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:27:05,731][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:27:06,062][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:27:06,383][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:27:06,708][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:27:07,033][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:27:07,368][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:27:07,685][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:27:08,012][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:27:08,338][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:27:08,664][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:27:08,989][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:27:09,314][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:27:09,640][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:27:10,393][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:27:11,095][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:27:11,096][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:27:11,097][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:27:12,021][__main__][INFO] - Iteration 382 took 23s (40.73% Gen, 55.39% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 26m 53s. Estimated total time: 19h 50m 17s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 40s, 500 more iterations: 3h 18m 22s.
[2025-11-13 10:27:12,023][__main__][INFO] - Starting iteration 382.
[2025-11-13 10:27:12,026][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:27:12,027][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:27:21,177][__main__][INFO] - Number of regex retries in iteration 382: 0
[2025-11-13 10:27:21,178][__main__][INFO] - agents played in iteration 382 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:27:21,633][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:21,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:21,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:21,733][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:21,734][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:27:21,735][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:27:22,483][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:27:22,778][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:27:23,102][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:27:23,428][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:27:23,753][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:27:24,077][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:27:24,402][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:27:24,727][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:27:25,052][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:27:25,378][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:27:25,704][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:27:26,030][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:27:26,355][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:27:26,682][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:27:27,013][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:27:27,337][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:27:27,663][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:27:27,988][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:27:28,318][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:27:28,646][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:27:28,970][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:27:29,297][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:27:29,628][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:27:29,954][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:27:30,280][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:27:30,605][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:27:30,931][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:27:31,256][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:27:31,583][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:27:31,907][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:27:32,234][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:27:32,559][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:27:32,884][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:27:33,645][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:27:34,357][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:27:34,359][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:27:34,361][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:27:35,329][__main__][INFO] - Iteration 383 took 23s (39.27% Gen, 56.57% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 1m 26s. Estimated total time: 19h 25m 12s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 50s, 500 more iterations: 3h 14m 12s.
[2025-11-13 10:27:35,331][__main__][INFO] - Starting iteration 383.
[2025-11-13 10:27:35,334][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:27:35,335][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:27:44,117][__main__][INFO] - Number of regex retries in iteration 383: 0
[2025-11-13 10:27:44,117][__main__][INFO] - agents played in iteration 383 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:27:44,591][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:44,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:45,011][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:45,045][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:27:45,046][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:27:45,046][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:27:45,795][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:27:46,090][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:27:46,414][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:27:46,740][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:27:47,071][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:27:47,389][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:27:47,715][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:27:48,040][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:27:48,371][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:27:48,688][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:27:49,013][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:27:49,337][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:27:49,669][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:27:49,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:27:50,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:27:50,649][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:27:50,980][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:27:51,304][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:27:51,635][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:27:51,961][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:27:52,290][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:27:52,616][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:27:52,942][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:27:53,268][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:27:53,593][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:27:53,918][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:27:54,245][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:27:54,571][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:27:54,898][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:27:55,222][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:27:55,547][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:27:55,873][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:27:56,198][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:27:56,976][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:27:57,711][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:27:57,712][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:27:57,713][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:27:58,640][__main__][INFO] - Iteration 384 took 23s (37.68% Gen, 58.33% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 1m 10s. Estimated total time: 19h 25m 20s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 50s, 500 more iterations: 3h 14m 13s.
[2025-11-13 10:27:58,642][__main__][INFO] - Starting iteration 384.
[2025-11-13 10:27:58,644][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:27:58,645][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:28:08,111][__main__][INFO] - Number of regex retries in iteration 384: 0
[2025-11-13 10:28:08,112][__main__][INFO] - agents played in iteration 384 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:28:08,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:08,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:08,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:08,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:08,672][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:28:08,673][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:28:09,424][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:28:09,720][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:28:10,045][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:28:10,369][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:28:10,696][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:28:11,019][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:28:11,343][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:28:11,667][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:28:11,996][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:28:12,318][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:28:12,643][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:28:12,969][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:28:13,300][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:28:13,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:28:13,950][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:28:14,275][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:28:14,605][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:28:14,929][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:28:15,257][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:28:15,587][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:28:15,915][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:28:16,241][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:28:16,568][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:28:16,894][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:28:17,220][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:28:17,546][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:28:17,872][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:28:18,197][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:28:18,522][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:28:18,850][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:28:19,177][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:28:19,506][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:28:19,833][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:28:20,618][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:28:21,320][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:28:21,322][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:28:21,324][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:28:22,238][__main__][INFO] - Iteration 385 took 23s (40.12% Gen, 56.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 15m 9s. Estimated total time: 19h 39m 43s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 19s, 500 more iterations: 3h 16m 37s.
[2025-11-13 10:28:22,240][__main__][INFO] - Starting iteration 385.
[2025-11-13 10:28:22,242][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:28:22,243][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:28:31,261][__main__][INFO] - Number of regex retries in iteration 385: 0
[2025-11-13 10:28:31,262][__main__][INFO] - agents played in iteration 385 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:28:31,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:31,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:31,799][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:31,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:31,833][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:28:31,834][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:28:32,577][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:28:32,872][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:28:33,197][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:28:33,522][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:28:33,846][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:28:34,171][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:28:34,495][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:28:34,821][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:28:35,151][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:28:35,477][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:28:35,803][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:28:36,126][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:28:36,456][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:28:36,784][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:28:37,109][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:28:37,441][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:28:37,770][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:28:38,097][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:28:38,423][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:28:38,750][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:28:39,077][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:28:39,403][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:28:39,730][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:28:40,063][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:28:40,388][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:28:40,714][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:28:41,039][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:28:41,369][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:28:41,697][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:28:42,023][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:28:42,348][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:28:42,676][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:28:43,002][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:28:43,768][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:28:44,493][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:28:44,495][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:28:44,496][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:28:45,543][__main__][INFO] - Iteration 386 took 23s (38.70% Gen, 56.79% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 9s. Estimated total time: 19h 25m 6s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 50s, 500 more iterations: 3h 14m 11s.
[2025-11-13 10:28:45,545][__main__][INFO] - Starting iteration 386.
[2025-11-13 10:28:45,548][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:28:45,549][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:28:54,589][__main__][INFO] - Number of regex retries in iteration 386: 0
[2025-11-13 10:28:54,589][__main__][INFO] - agents played in iteration 386 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:28:55,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:55,076][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:55,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:55,143][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:28:55,144][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:28:55,144][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:28:55,890][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:28:56,184][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:28:56,509][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:28:56,834][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:28:57,159][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:28:57,484][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:28:57,809][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:28:58,137][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:28:58,465][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:28:58,791][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:28:59,120][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:28:59,445][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:28:59,775][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:29:00,102][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:29:00,430][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:29:00,756][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:29:01,085][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:29:01,412][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:29:01,736][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:29:02,063][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:29:02,391][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:29:02,716][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:29:03,040][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:29:03,367][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:29:03,699][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:29:04,017][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:29:04,343][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:29:04,668][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:29:05,002][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:29:05,320][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:29:05,646][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:29:05,974][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:29:06,307][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:29:07,052][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:29:07,777][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:29:07,778][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:29:07,780][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:29:08,713][__main__][INFO] - Iteration 387 took 23s (39.02% Gen, 56.94% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 52m 58s. Estimated total time: 19h 18m 18s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 3s.
[2025-11-13 10:29:08,715][__main__][INFO] - Starting iteration 387.
[2025-11-13 10:29:08,719][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:29:08,719][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:29:18,858][__main__][INFO] - Number of regex retries in iteration 387: 0
[2025-11-13 10:29:18,858][__main__][INFO] - agents played in iteration 387 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:29:19,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:19,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:19,378][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:19,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:19,412][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:29:19,413][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:29:20,152][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:29:20,447][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:29:20,773][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:29:21,098][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:29:21,422][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:29:21,746][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:29:22,072][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:29:22,402][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:29:22,726][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:29:23,049][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:29:23,374][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:29:23,705][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:29:24,035][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:29:24,360][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:29:24,686][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:29:25,012][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:29:25,336][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:29:25,661][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:29:25,986][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:29:26,312][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:29:26,638][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:29:26,963][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:29:27,288][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:29:27,613][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:29:27,938][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:29:28,263][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:29:28,588][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:29:28,917][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:29:29,244][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:29:29,573][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:29:29,908][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:29:30,232][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:29:30,559][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:29:31,301][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:29:31,994][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:29:31,995][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:29:31,997][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:29:32,915][__main__][INFO] - Iteration 388 took 24s (41.90% Gen, 54.30% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 44m 7s. Estimated total time: 20h 9m 51s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 19s, 500 more iterations: 3h 21m 38s.
[2025-11-13 10:29:32,917][__main__][INFO] - Starting iteration 388.
[2025-11-13 10:29:32,921][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:29:32,921][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:29:42,358][__main__][INFO] - Number of regex retries in iteration 388: 0
[2025-11-13 10:29:42,358][__main__][INFO] - agents played in iteration 388 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:29:42,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:42,849][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:42,882][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:42,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:29:42,916][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:29:42,917][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:29:43,673][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:29:43,968][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:29:44,293][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:29:44,618][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:29:44,946][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:29:45,272][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:29:45,598][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:29:45,925][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:29:46,249][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:29:46,583][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:29:46,910][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:29:47,235][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:29:47,561][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:29:47,892][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:29:48,218][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:29:48,544][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:29:48,869][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:29:49,194][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:29:49,520][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:29:49,844][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:29:50,170][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:29:50,500][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:29:50,823][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:29:51,150][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:29:51,474][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:29:51,799][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:29:52,126][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:29:52,451][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:29:52,777][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:29:53,112][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:29:53,441][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:29:53,770][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:29:54,095][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:29:54,866][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:29:55,583][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:29:55,585][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:29:55,586][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:29:56,500][__main__][INFO] - Iteration 389 took 23s (40.02% Gen, 56.10% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 12m 52s. Estimated total time: 19h 39m 0s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 18s, 500 more iterations: 3h 16m 30s.
[2025-11-13 10:29:56,502][__main__][INFO] - Starting iteration 389.
[2025-11-13 10:29:56,505][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:29:56,506][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:30:05,728][__main__][INFO] - Number of regex retries in iteration 389: 0
[2025-11-13 10:30:05,729][__main__][INFO] - agents played in iteration 389 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:30:06,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:06,236][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:06,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:06,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:06,303][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:30:06,304][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:30:07,046][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:30:07,342][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:30:07,667][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:30:07,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:30:08,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:30:08,642][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:30:08,966][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:30:09,290][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:30:09,614][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:30:09,941][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:30:10,267][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:30:10,592][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:30:10,917][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:30:11,246][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:30:11,577][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:30:11,902][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:30:12,228][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:30:12,553][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:30:12,876][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:30:13,202][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:30:13,529][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:30:13,853][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:30:14,178][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:30:14,503][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:30:14,827][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:30:15,152][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:30:15,477][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:30:15,804][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:30:16,130][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:30:16,456][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:30:16,781][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:30:17,106][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:30:17,434][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:30:18,207][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:30:18,905][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:30:18,907][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:30:18,908][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:30:19,849][__main__][INFO] - Iteration 390 took 23s (39.51% Gen, 56.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 42s. Estimated total time: 19h 27m 13s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 54s, 500 more iterations: 3h 14m 32s.
[2025-11-13 10:30:19,851][__main__][INFO] - Starting iteration 390.
[2025-11-13 10:30:19,854][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 38 and human policies 1.
[2025-11-13 10:30:19,854][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:30:28,396][__main__][INFO] - Number of regex retries in iteration 390: 0
[2025-11-13 10:30:28,396][__main__][INFO] - agents played in iteration 390 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:30:28,860][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:28,893][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:28,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:28,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:28,960][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:30:28,961][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:30:29,721][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:30:30,018][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:30:30,342][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:30:30,667][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:30:30,992][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:30:31,320][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:30:31,645][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:30:31,970][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:30:32,298][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:30:32,626][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:30:32,954][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:30:33,279][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:30:33,604][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:30:33,931][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:30:34,258][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:30:34,585][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:30:34,910][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:30:35,239][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:30:35,564][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:30:35,888][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:30:36,213][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:30:36,538][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:30:36,864][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:30:37,190][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:30:37,515][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:30:37,846][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:30:38,173][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:30:38,499][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:30:38,825][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:30:39,154][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:30:39,480][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:30:39,804][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:30:40,130][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:30:40,892][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:30:41,624][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:30:41,626][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:30:41,629][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:30:43,569][__main__][INFO] - Iteration 391 took 23s (36.02% Gen, 55.80% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 18m 53s. Estimated total time: 19h 45m 48s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 31s, 500 more iterations: 3h 17m 38s.
[2025-11-13 10:30:43,571][__main__][INFO] - Starting iteration 391.
[2025-11-13 10:30:43,574][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:30:43,575][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:30:52,612][__main__][INFO] - Number of regex retries in iteration 391: 0
[2025-11-13 10:30:52,613][__main__][INFO] - agents played in iteration 391 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:30:53,072][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:53,106][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:53,139][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:53,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:30:53,172][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:30:53,173][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:30:53,941][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:30:54,237][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:30:54,563][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:30:54,888][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:30:55,214][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:30:55,539][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:30:55,864][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:30:56,189][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:30:56,517][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:30:56,844][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:30:57,169][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:30:57,496][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:30:57,822][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:30:58,146][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:30:58,470][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:30:58,795][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:30:59,119][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:30:59,444][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:30:59,768][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:31:00,092][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:31:00,418][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:31:00,742][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:31:01,067][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:31:01,393][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:31:01,722][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:31:02,048][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:31:02,373][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:31:02,704][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:31:03,029][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:31:03,355][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:31:03,682][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:31:04,010][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:31:04,335][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:31:05,093][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:31:05,830][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:31:05,831][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:31:05,833][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:31:06,752][__main__][INFO] - Iteration 392 took 23s (38.99% Gen, 57.04% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 38s. Estimated total time: 19h 18m 56s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 9s.
[2025-11-13 10:31:06,754][__main__][INFO] - Starting iteration 392.
[2025-11-13 10:31:06,758][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:31:06,759][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:31:15,869][__main__][INFO] - Number of regex retries in iteration 392: 0
[2025-11-13 10:31:15,870][__main__][INFO] - agents played in iteration 392 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:31:16,338][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:16,372][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:16,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:16,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:16,439][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:31:16,439][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:31:17,207][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:31:17,502][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:31:17,827][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:31:18,153][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:31:18,479][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:31:18,804][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:31:19,129][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:31:19,454][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:31:19,778][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:31:20,102][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:31:20,426][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:31:20,750][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:31:21,080][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:31:21,404][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:31:21,728][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:31:22,057][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:31:22,382][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:31:22,706][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:31:23,031][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:31:23,356][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:31:23,681][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:31:24,007][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:31:24,331][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:31:24,657][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:31:24,982][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:31:25,308][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:31:25,633][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:31:25,958][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:31:26,284][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:31:26,609][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:31:26,934][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:31:27,259][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:31:27,582][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:31:28,330][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:31:29,079][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:31:29,081][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:31:29,083][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:31:30,047][__main__][INFO] - Iteration 393 took 23s (39.12% Gen, 56.73% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 48s. Estimated total time: 19h 24m 29s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 4s.
[2025-11-13 10:31:30,050][__main__][INFO] - Starting iteration 393.
[2025-11-13 10:31:30,053][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:31:30,054][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:31:38,768][__main__][INFO] - Number of regex retries in iteration 393: 0
[2025-11-13 10:31:38,769][__main__][INFO] - agents played in iteration 393 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:31:39,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:39,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:39,312][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:39,346][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:31:39,347][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:31:39,347][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:31:40,124][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:31:40,421][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:31:40,746][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:31:41,070][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:31:41,395][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:31:41,719][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:31:42,052][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:31:42,372][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:31:42,696][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:31:43,020][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:31:43,349][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:31:43,674][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:31:43,999][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:31:44,323][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:31:44,648][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:31:44,974][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:31:45,299][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:31:45,624][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:31:45,956][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:31:46,277][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:31:46,600][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:31:46,926][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:31:47,254][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:31:47,579][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:31:47,903][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:31:48,231][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:31:48,556][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:31:48,881][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:31:49,206][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:31:49,531][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:31:49,858][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:31:50,182][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:31:50,507][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:31:51,230][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:31:51,956][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:31:51,957][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:31:51,959][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:31:53,012][__main__][INFO] - Iteration 394 took 22s (37.95% Gen, 57.45% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 39m 57s. Estimated total time: 19h 8m 1s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 16s, 500 more iterations: 3h 11m 20s.
[2025-11-13 10:31:53,014][__main__][INFO] - Starting iteration 394.
[2025-11-13 10:31:53,018][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:31:53,018][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:32:01,906][__main__][INFO] - Number of regex retries in iteration 394: 0
[2025-11-13 10:32:01,907][__main__][INFO] - agents played in iteration 394 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:32:02,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:02,405][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:02,437][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:02,470][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:02,471][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:32:02,471][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:32:03,265][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:32:03,560][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:32:03,885][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:32:04,209][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:32:04,535][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:32:04,862][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:32:05,190][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:32:05,515][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:32:05,841][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:32:06,166][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:32:06,489][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:32:06,814][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:32:07,146][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:32:07,471][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:32:07,800][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:32:08,129][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:32:08,463][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:32:08,792][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:32:09,116][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:32:09,442][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:32:09,767][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:32:10,093][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:32:10,418][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:32:10,743][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:32:11,068][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:32:11,393][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:32:11,719][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:32:12,044][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:32:12,371][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:32:12,698][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:32:13,026][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:32:13,350][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:32:13,677][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:32:14,413][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:32:15,174][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:32:15,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:32:15,178][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:32:16,169][__main__][INFO] - Iteration 395 took 23s (38.39% Gen, 57.32% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 49m 8s. Estimated total time: 19h 17m 36s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 35s, 500 more iterations: 3h 12m 56s.
[2025-11-13 10:32:16,171][__main__][INFO] - Starting iteration 395.
[2025-11-13 10:32:16,175][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:32:16,175][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:32:25,543][__main__][INFO] - Number of regex retries in iteration 395: 0
[2025-11-13 10:32:25,544][__main__][INFO] - agents played in iteration 395 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:32:26,007][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:26,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:26,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:26,108][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:26,108][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:32:26,109][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:32:26,888][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:32:27,184][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:32:27,510][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:32:27,834][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:32:28,159][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:32:28,483][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:32:28,810][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:32:29,136][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:32:29,467][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:32:29,788][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:32:30,113][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:32:30,439][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:32:30,767][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:32:31,094][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:32:31,422][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:32:31,750][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:32:32,080][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:32:32,410][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:32:32,738][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:32:33,064][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:32:33,391][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:32:33,717][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:32:34,042][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:32:34,370][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:32:34,696][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:32:35,023][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:32:35,349][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:32:35,674][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:32:35,999][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:32:36,325][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:32:36,650][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:32:36,976][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:32:37,301][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:32:38,029][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:32:38,773][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:32:38,775][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:32:38,777][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:32:39,792][__main__][INFO] - Iteration 396 took 23s (39.67% Gen, 56.03% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 12m 4s. Estimated total time: 19h 40m 55s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 21s, 500 more iterations: 3h 16m 49s.
[2025-11-13 10:32:39,794][__main__][INFO] - Starting iteration 396.
[2025-11-13 10:32:39,798][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:32:39,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:32:48,936][__main__][INFO] - Number of regex retries in iteration 396: 0
[2025-11-13 10:32:48,937][__main__][INFO] - agents played in iteration 396 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:32:49,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:49,430][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:49,463][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:49,497][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:32:49,497][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:32:49,498][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:32:50,248][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:32:50,543][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:32:50,868][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:32:51,194][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:32:51,519][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:32:51,845][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:32:52,171][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:32:52,496][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:32:52,821][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:32:53,149][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:32:53,475][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:32:53,800][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:32:54,130][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:32:54,455][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:32:54,782][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:32:55,112][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:32:55,439][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:32:55,768][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:32:56,093][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:32:56,418][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:32:56,744][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:32:57,068][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:32:57,393][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:32:57,720][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:32:58,047][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:32:58,377][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:32:58,698][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:32:59,023][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:32:59,348][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:32:59,674][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:33:00,000][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:33:00,326][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:33:00,655][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:33:01,378][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:33:02,107][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:33:02,109][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:33:02,110][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:33:03,158][__main__][INFO] - Iteration 397 took 23s (39.12% Gen, 56.39% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 58m 48s. Estimated total time: 19h 28m 3s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 40s.
[2025-11-13 10:33:03,160][__main__][INFO] - Starting iteration 397.
[2025-11-13 10:33:03,163][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:33:03,164][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:33:12,687][__main__][INFO] - Number of regex retries in iteration 397: 0
[2025-11-13 10:33:12,688][__main__][INFO] - agents played in iteration 397 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:33:13,152][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:13,185][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:13,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:13,252][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:13,253][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:33:13,254][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:33:14,018][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:33:14,316][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:33:14,642][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:33:14,966][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:33:15,291][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:33:15,616][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:33:15,940][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:33:16,267][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:33:16,592][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:33:16,919][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:33:17,244][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:33:17,573][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:33:17,897][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:33:18,224][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:33:18,550][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:33:18,876][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:33:19,203][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:33:19,538][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:33:19,863][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:33:20,188][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:33:20,514][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:33:20,851][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:33:21,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:33:21,500][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:33:21,826][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:33:22,157][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:33:22,483][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:33:22,809][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:33:23,134][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:33:23,470][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:33:23,799][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:33:24,129][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:33:24,461][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:33:25,193][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:33:25,945][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:33:25,946][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:33:25,948][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:33:27,315][__main__][INFO] - Iteration 398 took 24s (39.43% Gen, 54.90% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 37m 59s. Estimated total time: 20h 7m 38s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 15s, 500 more iterations: 3h 21m 16s.
[2025-11-13 10:33:27,317][__main__][INFO] - Starting iteration 398.
[2025-11-13 10:33:27,321][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:33:27,321][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:33:36,438][__main__][INFO] - Number of regex retries in iteration 398: 0
[2025-11-13 10:33:36,438][__main__][INFO] - agents played in iteration 398 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:33:36,903][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:36,937][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:36,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:37,004][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:33:37,005][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:33:37,005][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:33:37,787][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:33:38,083][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:33:38,408][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:33:38,732][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:33:39,064][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:33:39,382][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:33:39,708][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:33:40,036][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:33:40,369][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:33:40,686][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:33:41,013][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:33:41,339][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:33:41,671][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:33:41,994][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:33:42,321][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:33:42,646][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:33:42,977][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:33:43,298][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:33:43,623][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:33:43,948][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:33:44,278][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:33:44,602][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:33:44,927][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:33:45,252][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:33:45,583][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:33:45,905][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:33:46,231][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:33:46,556][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:33:46,884][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:33:47,210][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:33:47,537][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:33:47,863][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:33:48,190][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:33:48,930][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:33:49,676][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:33:49,677][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:33:49,679][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:33:50,665][__main__][INFO] - Iteration 399 took 23s (39.05% Gen, 56.72% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 57m 15s. Estimated total time: 19h 27m 17s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 54s, 500 more iterations: 3h 14m 32s.
[2025-11-13 10:33:50,668][__main__][INFO] - Starting iteration 399.
[2025-11-13 10:33:50,671][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:33:50,671][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:34:00,066][__main__][INFO] - Number of regex retries in iteration 399: 0
[2025-11-13 10:34:00,067][__main__][INFO] - agents played in iteration 399 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:34:00,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:00,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:00,615][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:00,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:00,649][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:34:00,649][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:34:01,417][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:34:01,714][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:34:02,040][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:34:02,365][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:34:02,691][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:34:03,017][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:34:03,346][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:34:03,672][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:34:03,998][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:34:04,324][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:34:04,649][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:34:04,974][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:34:05,299][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:34:05,625][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:34:05,956][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:34:06,282][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:34:06,608][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:34:06,939][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:34:07,266][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:34:07,592][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:34:07,918][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:34:08,253][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:34:08,578][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:34:08,904][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:34:09,230][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:34:09,563][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:34:09,888][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:34:10,213][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:34:10,539][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:34:10,867][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:34:11,192][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:34:11,517][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:34:11,842][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:34:12,574][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:34:13,311][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:34:13,313][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:34:13,315][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:34:14,436][__main__][INFO] - Iteration 400 took 23s (39.53% Gen, 55.74% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 17m 52s. Estimated total time: 19h 48m 17s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 36s, 500 more iterations: 3h 18m 2s.
[2025-11-13 10:34:14,438][__main__][INFO] - Starting iteration 400.
[2025-11-13 10:34:14,441][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 39 and human policies 1.
[2025-11-13 10:34:14,442][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:34:24,085][__main__][INFO] - Number of regex retries in iteration 400: 0
[2025-11-13 10:34:24,086][__main__][INFO] - agents played in iteration 400 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:34:24,542][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,576][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,610][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:24,644][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:34:24,644][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:34:25,431][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:34:25,729][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:34:26,056][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:34:26,382][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:34:26,710][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:34:27,036][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:34:27,360][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:34:27,689][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:34:28,015][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:34:28,340][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:34:28,665][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:34:28,991][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:34:29,317][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:34:29,642][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:34:29,968][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:34:30,296][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:34:30,620][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:34:30,945][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:34:31,272][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:34:31,605][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:34:31,925][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:34:32,250][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:34:32,575][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:34:32,903][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:34:33,227][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:34:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:34:33,876][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:34:34,203][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:34:34,525][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:34:34,850][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:34:35,178][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:34:35,507][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:34:35,829][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:34:36,574][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:34:37,300][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:34:37,302][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:34:37,304][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:34:39,344][__main__][INFO] - Iteration 401 took 24s (38.73% Gen, 53.08% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 14m 19s. Estimated total time: 20h 45m 10s. Time estimates for 10 more iterations: 4m 9s, 100 more iterations: 41m 30s, 500 more iterations: 3h 27m 31s.
[2025-11-13 10:34:39,346][__main__][INFO] - Starting iteration 401.
[2025-11-13 10:34:39,349][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:34:39,349][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:34:49,237][__main__][INFO] - Number of regex retries in iteration 401: 0
[2025-11-13 10:34:49,237][__main__][INFO] - agents played in iteration 401 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:34:49,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:49,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:49,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:49,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:34:49,817][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:34:49,817][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:34:50,598][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:34:50,893][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:34:51,220][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:34:51,545][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:34:51,870][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:34:52,194][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:34:52,520][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:34:52,845][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:34:53,170][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:34:53,495][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:34:53,821][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:34:54,147][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:34:54,475][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:34:54,801][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:34:55,127][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:34:55,454][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:34:55,781][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:34:56,110][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:34:56,436][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:34:56,761][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:34:57,087][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:34:57,412][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:34:57,737][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:34:58,065][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:34:58,390][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:34:58,718][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:34:59,043][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:34:59,371][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:34:59,697][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:35:00,022][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:35:00,350][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:35:00,676][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:35:01,003][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:35:01,782][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:35:02,548][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:35:02,549][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:35:02,551][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:35:03,524][__main__][INFO] - Iteration 402 took 24s (40.90% Gen, 55.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 37m 33s. Estimated total time: 20h 8m 48s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 17s, 500 more iterations: 3h 21m 28s.
[2025-11-13 10:35:03,526][__main__][INFO] - Starting iteration 402.
[2025-11-13 10:35:03,529][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:35:03,530][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:35:12,413][__main__][INFO] - Number of regex retries in iteration 402: 0
[2025-11-13 10:35:12,413][__main__][INFO] - agents played in iteration 402 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:35:12,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,917][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,951][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,985][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:12,985][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:35:12,986][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:35:13,782][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:35:14,077][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:35:14,402][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:35:14,726][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:35:15,052][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:35:15,378][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:35:15,704][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:35:16,028][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:35:16,352][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:35:16,678][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:35:17,004][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:35:17,333][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:35:17,661][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:35:17,986][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:35:18,311][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:35:18,636][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:35:18,961][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:35:19,288][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:35:19,613][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:35:19,939][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:35:20,264][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:35:20,593][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:35:20,918][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:35:21,244][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:35:21,569][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:35:21,898][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:35:22,223][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:35:22,547][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:35:22,873][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:35:23,201][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:35:23,527][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:35:23,855][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:35:24,183][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:35:24,946][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:35:25,699][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:35:25,700][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:35:25,702][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:35:26,689][__main__][INFO] - Iteration 403 took 23s (38.35% Gen, 57.38% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 46m 25s. Estimated total time: 19h 18m 3s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 0s.
[2025-11-13 10:35:26,691][__main__][INFO] - Starting iteration 403.
[2025-11-13 10:35:26,695][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:35:26,695][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:35:36,015][__main__][INFO] - Number of regex retries in iteration 403: 0
[2025-11-13 10:35:36,016][__main__][INFO] - agents played in iteration 403 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:35:36,492][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:36,525][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:36,559][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:36,592][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:36,593][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:35:36,593][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:35:37,407][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:35:37,705][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:35:38,030][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:35:38,356][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:35:38,682][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:35:39,006][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:35:39,331][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:35:39,656][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:35:39,983][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:35:40,308][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:35:40,636][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:35:40,962][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:35:41,298][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:35:41,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:35:41,949][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:35:42,273][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:35:42,606][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:35:42,924][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:35:43,249][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:35:43,575][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:35:43,908][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:35:44,226][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:35:44,555][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:35:44,881][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:35:45,214][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:35:45,534][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:35:45,863][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:35:46,189][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:35:46,518][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:35:46,838][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:35:47,165][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:35:47,492][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:35:47,823][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:35:48,568][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:35:49,325][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:35:49,327][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:35:49,328][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:35:50,400][__main__][INFO] - Iteration 404 took 23s (39.32% Gen, 56.16% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 13m 16s. Estimated total time: 19h 45m 18s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 33s.
[2025-11-13 10:35:50,403][__main__][INFO] - Starting iteration 404.
[2025-11-13 10:35:50,406][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:35:50,407][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:35:59,213][__main__][INFO] - Number of regex retries in iteration 404: 0
[2025-11-13 10:35:59,214][__main__][INFO] - agents played in iteration 404 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:35:59,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:59,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:59,773][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:59,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:35:59,821][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:35:59,821][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:36:00,623][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:36:00,921][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:36:01,249][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:36:01,573][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:36:01,899][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:36:02,223][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:36:02,548][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:36:02,881][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:36:03,206][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:36:03,530][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:36:03,855][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:36:04,181][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:36:04,506][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:36:04,832][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:36:05,156][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:36:05,489][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:36:05,816][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:36:06,141][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:36:06,466][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:36:06,790][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:36:07,114][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:36:07,440][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:36:07,766][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:36:08,101][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:36:08,425][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:36:08,750][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:36:09,074][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:36:09,404][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:36:09,729][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:36:10,059][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:36:10,383][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:36:10,710][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:36:11,038][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:36:11,808][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:36:12,536][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:36:12,537][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:36:12,539][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:36:13,595][__main__][INFO] - Iteration 405 took 23s (37.98% Gen, 57.46% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 47m 3s. Estimated total time: 19h 19m 28s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 38s, 500 more iterations: 3h 13m 14s.
[2025-11-13 10:36:13,597][__main__][INFO] - Starting iteration 405.
[2025-11-13 10:36:13,601][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:36:13,601][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:36:23,151][__main__][INFO] - Number of regex retries in iteration 405: 0
[2025-11-13 10:36:23,152][__main__][INFO] - agents played in iteration 405 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:36:23,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:23,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:23,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:23,738][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:23,739][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:36:23,740][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:36:24,527][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:36:24,824][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:36:25,149][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:36:25,473][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:36:25,799][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:36:26,125][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:36:26,449][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:36:26,775][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:36:27,100][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:36:27,426][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:36:27,750][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:36:28,074][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:36:28,399][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:36:28,732][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:36:29,053][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:36:29,378][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:36:29,704][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:36:30,035][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:36:30,358][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:36:30,684][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:36:31,011][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:36:31,338][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:36:31,663][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:36:31,987][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:36:32,312][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:36:32,640][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:36:32,963][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:36:33,288][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:36:33,613][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:36:33,940][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:36:34,265][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:36:34,593][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:36:34,921][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:36:35,646][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:36:36,367][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:36:36,369][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:36:36,371][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:36:37,503][__main__][INFO] - Iteration 406 took 23s (39.96% Gen, 55.30% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 22m 21s. Estimated total time: 19h 55m 10s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 11s.
[2025-11-13 10:36:37,505][__main__][INFO] - Starting iteration 406.
[2025-11-13 10:36:37,509][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:36:37,510][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:36:46,210][__main__][INFO] - Number of regex retries in iteration 406: 0
[2025-11-13 10:36:46,211][__main__][INFO] - agents played in iteration 406 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:36:46,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:46,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:46,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:46,778][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:36:46,778][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:36:46,779][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:36:47,561][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:36:47,857][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:36:48,183][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:36:48,509][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:36:48,834][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:36:49,161][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:36:49,487][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:36:49,817][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:36:50,142][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:36:50,466][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:36:50,790][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:36:51,121][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:36:51,446][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:36:51,770][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:36:52,096][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:36:52,426][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:36:52,757][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:36:53,084][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:36:53,414][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:36:53,738][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:36:54,064][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:36:54,391][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:36:54,725][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:36:55,049][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:36:55,377][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:36:55,705][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:36:56,035][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:36:56,354][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:36:56,685][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:36:57,013][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:36:57,345][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:36:57,670][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:36:58,001][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:36:58,722][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:36:59,447][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:36:59,449][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:36:59,451][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:37:00,585][__main__][INFO] - Iteration 407 took 23s (37.70% Gen, 57.37% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 40m 38s. Estimated total time: 19h 13m 50s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 27s, 500 more iterations: 3h 12m 18s.
[2025-11-13 10:37:00,587][__main__][INFO] - Starting iteration 407.
[2025-11-13 10:37:00,590][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:37:00,591][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:37:08,858][__main__][INFO] - Number of regex retries in iteration 407: 0
[2025-11-13 10:37:08,858][__main__][INFO] - agents played in iteration 407 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:37:09,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:09,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:09,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:09,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:09,791][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:37:09,791][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:37:10,580][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:37:10,877][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:37:11,203][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:37:11,529][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:37:11,852][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:37:12,177][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:37:12,502][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:37:12,827][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:37:13,151][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:37:13,476][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:37:13,801][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:37:14,126][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:37:14,452][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:37:14,777][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:37:15,102][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:37:15,427][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:37:15,752][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:37:16,077][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:37:16,402][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:37:16,727][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:37:17,051][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:37:17,377][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:37:17,701][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:37:18,025][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:37:18,353][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:37:18,679][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:37:19,005][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:37:19,329][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:37:19,656][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:37:19,980][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:37:20,305][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:37:20,630][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:37:20,955][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:37:21,710][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:37:22,461][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:37:22,463][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:37:22,464][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:37:23,484][__main__][INFO] - Iteration 408 took 22s (36.11% Gen, 59.43% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 31m 8s. Estimated total time: 19h 4m 43s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 9s, 500 more iterations: 3h 10m 47s.
[2025-11-13 10:37:23,486][__main__][INFO] - Starting iteration 408.
[2025-11-13 10:37:23,489][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:37:23,490][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:37:32,564][__main__][INFO] - Number of regex retries in iteration 408: 0
[2025-11-13 10:37:32,565][__main__][INFO] - agents played in iteration 408 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:37:33,047][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:33,080][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:33,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:33,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:33,148][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:37:33,149][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:37:33,917][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:37:34,213][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:37:34,539][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:37:34,863][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:37:35,189][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:37:35,513][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:37:35,838][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:37:36,164][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:37:36,489][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:37:36,813][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:37:37,139][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:37:37,463][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:37:37,788][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:37:38,112][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:37:38,442][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:37:38,766][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:37:39,094][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:37:39,420][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:37:39,746][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:37:40,072][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:37:40,397][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:37:40,726][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:37:41,053][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:37:41,382][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:37:41,710][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:37:42,043][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:37:42,361][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:37:42,686][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:37:43,013][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:37:43,347][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:37:43,668][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:37:43,997][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:37:44,327][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:37:45,075][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:37:45,813][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:37:45,815][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:37:45,816][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:37:46,803][__main__][INFO] - Iteration 409 took 23s (38.92% Gen, 56.83% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 47s. Estimated total time: 19h 25m 44s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 51s, 500 more iterations: 3h 14m 17s.
[2025-11-13 10:37:46,806][__main__][INFO] - Starting iteration 409.
[2025-11-13 10:37:46,808][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:37:46,809][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:37:56,407][__main__][INFO] - Number of regex retries in iteration 409: 0
[2025-11-13 10:37:56,407][__main__][INFO] - agents played in iteration 409 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:37:56,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:56,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:56,943][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:56,976][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:37:56,977][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:37:56,977][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:37:57,752][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:37:58,049][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:37:58,374][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:37:58,699][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:37:59,023][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:37:59,353][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:37:59,678][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:38:00,003][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:38:00,328][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:38:00,655][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:38:00,983][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:38:01,307][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:38:01,635][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:38:01,960][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:38:02,285][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:38:02,610][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:38:02,936][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:38:03,262][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:38:03,587][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:38:03,912][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:38:04,237][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:38:04,562][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:38:04,889][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:38:05,215][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:38:05,541][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:38:05,864][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:38:06,193][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:38:06,517][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:38:06,844][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:38:07,166][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:38:07,491][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:38:07,816][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:38:08,142][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:38:08,896][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:38:09,628][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:38:09,629][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:38:09,631][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:38:10,640][__main__][INFO] - Iteration 410 took 23s (40.27% Gen, 55.49% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 17m 16s. Estimated total time: 19h 51m 38s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 36s.
[2025-11-13 10:38:10,642][__main__][INFO] - Starting iteration 410.
[2025-11-13 10:38:10,646][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 40 and human policies 1.
[2025-11-13 10:38:10,646][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:38:19,961][__main__][INFO] - Number of regex retries in iteration 410: 0
[2025-11-13 10:38:19,962][__main__][INFO] - agents played in iteration 410 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:38:20,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:20,459][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:20,492][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:20,526][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:20,526][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:38:20,527][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:38:21,305][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:38:21,601][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:38:21,933][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:38:22,258][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:38:22,584][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:38:22,909][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:38:23,246][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:38:23,571][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:38:23,895][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:38:24,220][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:38:24,552][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:38:24,877][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:38:25,202][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:38:25,528][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:38:25,855][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:38:26,181][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:38:26,507][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:38:26,832][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:38:27,161][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:38:27,485][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:38:27,810][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:38:28,136][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:38:28,462][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:38:28,785][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:38:29,109][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:38:29,437][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:38:29,764][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:38:30,091][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:38:30,418][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:38:30,752][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:38:31,071][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:38:31,402][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:38:31,727][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:38:32,476][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:38:33,191][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:38:33,192][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:38:33,194][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:38:35,197][__main__][INFO] - Iteration 411 took 24s (37.94% Gen, 53.90% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 52m 49s. Estimated total time: 20h 27m 36s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 55s, 500 more iterations: 3h 24m 36s.
[2025-11-13 10:38:35,199][__main__][INFO] - Starting iteration 411.
[2025-11-13 10:38:35,203][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:38:35,203][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:38:44,859][__main__][INFO] - Number of regex retries in iteration 411: 0
[2025-11-13 10:38:44,859][__main__][INFO] - agents played in iteration 411 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:38:45,341][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:45,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:45,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:45,477][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:38:45,477][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:38:45,477][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:38:46,266][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:38:46,560][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:38:46,885][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:38:47,210][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:38:47,535][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:38:47,862][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:38:48,187][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:38:48,512][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:38:48,837][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:38:49,162][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:38:49,486][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:38:49,810][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:38:50,135][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:38:50,465][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:38:50,793][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:38:51,123][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:38:51,455][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:38:51,782][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:38:52,115][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:38:52,441][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:38:52,768][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:38:53,093][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:38:53,417][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:38:53,745][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:38:54,071][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:38:54,397][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:38:54,726][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:38:55,056][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:38:55,383][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:38:55,707][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:38:56,038][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:38:56,364][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:38:56,688][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:38:57,447][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:38:58,175][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:38:58,176][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:38:58,178][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:38:59,173][__main__][INFO] - Iteration 412 took 23s (40.28% Gen, 55.56% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 23m 24s. Estimated total time: 19h 58m 35s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 57s, 500 more iterations: 3h 19m 45s.
[2025-11-13 10:38:59,176][__main__][INFO] - Starting iteration 412.
[2025-11-13 10:38:59,178][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:38:59,179][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:39:08,878][__main__][INFO] - Number of regex retries in iteration 412: 0
[2025-11-13 10:39:08,879][__main__][INFO] - agents played in iteration 412 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:39:09,373][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:09,407][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:09,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:09,475][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:09,476][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:39:09,476][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:39:10,245][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:39:10,544][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:39:10,872][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:39:11,196][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:39:11,522][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:39:11,846][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:39:12,172][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:39:12,499][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:39:12,824][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:39:13,155][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:39:13,477][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:39:13,803][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:39:14,130][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:39:14,459][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:39:14,780][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:39:15,107][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:39:15,432][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:39:15,766][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:39:16,085][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:39:16,410][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:39:16,737][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:39:17,062][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:39:17,387][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:39:17,711][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:39:18,038][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:39:18,362][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:39:18,686][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:39:19,014][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:39:19,339][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:39:19,668][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:39:19,995][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:39:20,319][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:39:20,646][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:39:21,392][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:39:22,122][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:39:22,124][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:39:22,126][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:39:23,157][__main__][INFO] - Iteration 413 took 23s (40.44% Gen, 55.24% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 23m 25s. Estimated total time: 19h 58m 59s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 57s, 500 more iterations: 3h 19m 49s.
[2025-11-13 10:39:23,160][__main__][INFO] - Starting iteration 413.
[2025-11-13 10:39:23,163][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:39:23,164][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:39:32,268][__main__][INFO] - Number of regex retries in iteration 413: 0
[2025-11-13 10:39:32,269][__main__][INFO] - agents played in iteration 413 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:39:32,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:32,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:32,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:32,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:32,838][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:39:32,838][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:39:33,644][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:39:33,939][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:39:34,272][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:39:34,588][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:39:34,913][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:39:35,238][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:39:35,562][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:39:35,888][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:39:36,214][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:39:36,539][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:39:36,864][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:39:37,190][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:39:37,515][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:39:37,839][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:39:38,166][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:39:38,492][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:39:38,819][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:39:39,144][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:39:39,470][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:39:39,796][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:39:40,121][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:39:40,448][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:39:40,774][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:39:41,101][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:39:41,428][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:39:41,756][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:39:42,084][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:39:42,411][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:39:42,737][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:39:43,064][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:39:43,390][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:39:43,714][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:39:44,041][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:39:44,743][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:39:45,476][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:39:45,478][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:39:45,479][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:39:46,507][__main__][INFO] - Iteration 414 took 23s (39.00% Gen, 56.59% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 17s. Estimated total time: 19h 27m 14s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 54s, 500 more iterations: 3h 14m 32s.
[2025-11-13 10:39:46,509][__main__][INFO] - Starting iteration 414.
[2025-11-13 10:39:46,513][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:39:46,513][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:39:55,335][__main__][INFO] - Number of regex retries in iteration 414: 0
[2025-11-13 10:39:55,336][__main__][INFO] - agents played in iteration 414 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:39:55,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:56,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:56,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:56,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:39:56,290][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:39:56,291][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:39:57,078][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:39:57,373][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:39:57,699][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:39:58,025][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:39:58,361][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:39:58,685][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:39:59,010][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:39:59,334][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:39:59,659][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:39:59,984][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:40:00,309][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:40:00,633][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:40:00,970][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:40:01,298][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:40:01,629][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:40:01,954][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:40:02,281][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:40:02,609][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:40:02,934][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:40:03,261][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:40:03,588][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:40:03,913][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:40:04,238][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:40:04,572][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:40:04,893][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:40:05,221][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:40:05,544][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:40:05,872][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:40:06,194][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:40:06,521][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:40:06,849][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:40:07,182][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:40:07,500][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:40:08,199][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:40:08,941][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:40:08,943][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:40:08,944][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:40:10,053][__main__][INFO] - Iteration 415 took 23s (37.48% Gen, 57.81% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 0m 40s. Estimated total time: 19h 37m 2s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 10s.
[2025-11-13 10:40:10,055][__main__][INFO] - Starting iteration 415.
[2025-11-13 10:40:10,058][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:40:10,058][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:40:18,893][__main__][INFO] - Number of regex retries in iteration 415: 0
[2025-11-13 10:40:18,894][__main__][INFO] - agents played in iteration 415 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:40:19,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:19,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:19,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:19,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:19,453][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:40:19,454][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:40:20,239][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:40:20,535][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:40:20,861][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:40:21,189][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:40:21,515][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:40:21,839][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:40:22,164][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:40:22,489][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:40:22,814][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:40:23,140][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:40:23,464][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:40:23,789][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:40:24,117][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:40:24,443][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:40:24,770][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:40:25,094][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:40:25,419][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:40:25,744][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:40:26,074][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:40:26,399][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:40:26,730][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:40:27,070][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:40:27,400][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:40:27,723][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:40:28,048][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:40:28,381][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:40:28,706][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:40:29,032][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:40:29,362][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:40:29,690][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:40:30,013][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:40:30,337][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:40:30,661][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:40:31,365][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:40:32,097][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:40:32,098][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:40:32,100][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:40:33,188][__main__][INFO] - Iteration 416 took 23s (38.19% Gen, 57.09% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 39m 48s. Estimated total time: 19h 16m 32s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 45s.
[2025-11-13 10:40:33,190][__main__][INFO] - Starting iteration 416.
[2025-11-13 10:40:33,194][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:40:33,194][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:40:42,840][__main__][INFO] - Number of regex retries in iteration 416: 0
[2025-11-13 10:40:42,841][__main__][INFO] - agents played in iteration 416 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:40:43,321][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:43,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:43,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:43,422][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:40:43,423][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:40:43,424][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:40:44,220][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:40:44,515][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:40:44,841][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:40:45,166][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:40:45,491][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:40:45,817][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:40:46,143][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:40:46,467][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:40:46,797][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:40:47,124][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:40:47,449][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:40:47,775][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:40:48,103][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:40:48,428][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:40:48,757][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:40:49,088][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:40:49,413][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:40:49,739][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:40:50,063][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:40:50,388][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:40:50,711][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:40:51,037][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:40:51,361][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:40:51,688][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:40:52,018][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:40:52,346][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:40:52,673][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:40:53,000][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:40:53,330][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:40:53,654][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:40:53,979][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:40:54,303][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:40:54,628][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:40:55,322][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:40:56,053][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:40:56,054][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:40:56,056][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:40:57,133][__main__][INFO] - Iteration 417 took 23s (40.29% Gen, 55.20% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 19m 52s. Estimated total time: 19h 57m 1s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 54s, 500 more iterations: 3h 19m 30s.
[2025-11-13 10:40:57,135][__main__][INFO] - Starting iteration 417.
[2025-11-13 10:40:57,139][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:40:57,139][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:41:06,083][__main__][INFO] - Number of regex retries in iteration 417: 0
[2025-11-13 10:41:06,084][__main__][INFO] - agents played in iteration 417 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:41:06,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:06,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:06,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:06,671][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:06,672][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:41:06,672][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:41:07,428][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:41:07,727][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:41:08,053][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:41:08,377][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:41:08,705][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:41:09,029][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:41:09,354][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:41:09,680][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:41:10,007][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:41:10,331][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:41:10,657][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:41:10,982][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:41:11,306][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:41:11,634][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:41:11,962][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:41:12,292][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:41:12,620][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:41:12,951][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:41:13,278][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:41:13,608][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:41:13,936][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:41:14,262][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:41:14,590][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:41:14,913][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:41:15,240][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:41:15,566][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:41:15,891][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:41:16,221][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:41:16,551][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:41:16,880][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:41:17,206][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:41:17,530][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:41:17,854][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:41:18,569][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:41:19,290][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:41:19,291][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:41:19,293][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:41:20,408][__main__][INFO] - Iteration 418 took 23s (38.44% Gen, 56.77% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 45m 58s. Estimated total time: 19h 23m 29s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 46s, 500 more iterations: 3h 13m 54s.
[2025-11-13 10:41:20,410][__main__][INFO] - Starting iteration 418.
[2025-11-13 10:41:20,414][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:41:20,415][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:41:29,529][__main__][INFO] - Number of regex retries in iteration 418: 0
[2025-11-13 10:41:29,530][__main__][INFO] - agents played in iteration 418 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:41:30,017][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:30,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:30,084][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:30,117][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:30,118][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:41:30,118][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:41:30,879][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:41:31,174][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:41:31,505][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:41:31,830][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:41:32,156][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:41:32,482][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:41:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:41:33,134][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:41:33,459][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:41:33,785][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:41:34,112][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:41:34,439][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:41:34,764][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:41:35,093][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:41:35,414][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:41:35,746][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:41:36,072][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:41:36,405][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:41:36,725][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:41:37,057][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:41:37,382][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:41:37,709][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:41:38,038][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:41:38,367][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:41:38,692][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:41:39,022][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:41:39,344][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:41:39,669][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:41:39,995][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:41:40,325][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:41:40,649][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:41:40,975][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:41:41,299][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:41:42,036][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:41:42,773][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:41:42,775][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:41:42,776][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:41:43,768][__main__][INFO] - Iteration 419 took 23s (39.03% Gen, 56.72% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 49m 50s. Estimated total time: 19h 27m 45s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 55s, 500 more iterations: 3h 14m 37s.
[2025-11-13 10:41:43,770][__main__][INFO] - Starting iteration 419.
[2025-11-13 10:41:43,774][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:41:43,774][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:41:53,114][__main__][INFO] - Number of regex retries in iteration 419: 0
[2025-11-13 10:41:53,115][__main__][INFO] - agents played in iteration 419 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:41:53,593][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:53,626][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:53,659][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:53,693][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:41:53,693][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:41:53,694][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:41:54,483][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:41:54,779][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:41:55,106][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:41:55,432][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:41:55,757][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:41:56,082][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:41:56,409][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:41:56,734][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:41:57,059][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:41:57,385][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:41:57,711][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:41:58,037][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:41:58,362][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:41:58,693][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:41:59,019][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:41:59,346][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:41:59,670][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:41:59,994][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:42:00,320][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:42:00,647][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:42:00,974][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:42:01,299][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:42:01,628][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:42:01,956][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:42:02,285][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:42:02,616][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:42:02,943][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:42:03,270][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:42:03,596][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:42:03,921][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:42:04,246][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:42:04,571][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:42:04,896][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:42:05,622][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:42:06,346][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:42:06,351][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:42:06,352][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:42:07,365][__main__][INFO] - Iteration 420 took 23s (39.59% Gen, 56.11% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 1m 18s. Estimated total time: 19h 39m 37s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 19s, 500 more iterations: 3h 16m 36s.
[2025-11-13 10:42:07,367][__main__][INFO] - Starting iteration 420.
[2025-11-13 10:42:07,371][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 41 and human policies 1.
[2025-11-13 10:42:07,372][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:42:16,862][__main__][INFO] - Number of regex retries in iteration 420: 0
[2025-11-13 10:42:16,863][__main__][INFO] - agents played in iteration 420 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:42:17,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:17,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:17,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:17,436][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:17,437][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:42:17,438][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:42:18,193][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:42:18,490][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:42:18,818][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:42:19,142][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:42:19,467][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:42:19,793][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:42:20,118][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:42:20,444][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:42:20,769][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:42:21,095][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:42:21,421][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:42:21,747][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:42:22,073][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:42:22,400][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:42:22,727][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:42:23,054][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:42:23,379][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:42:23,706][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:42:24,033][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:42:24,359][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:42:24,687][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:42:25,017][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:42:25,342][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:42:25,670][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:42:25,996][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:42:26,321][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:42:26,649][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:42:26,979][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:42:27,307][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:42:27,631][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:42:27,958][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:42:28,286][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:42:28,612][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:42:29,316][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:42:30,039][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:42:30,040][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:42:30,042][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:42:32,046][__main__][INFO] - Iteration 421 took 24s (38.46% Gen, 53.41% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 55m 7s. Estimated total time: 20h 33m 50s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 7s, 500 more iterations: 3h 25m 38s.
[2025-11-13 10:42:32,048][__main__][INFO] - Starting iteration 421.
[2025-11-13 10:42:32,052][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:42:32,052][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:42:40,948][__main__][INFO] - Number of regex retries in iteration 421: 0
[2025-11-13 10:42:40,949][__main__][INFO] - agents played in iteration 421 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:42:41,412][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:41,446][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:41,480][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:41,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:42:41,514][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:42:41,514][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:42:42,287][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:42:42,584][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:42:42,910][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:42:43,235][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:42:43,561][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:42:43,886][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:42:44,211][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:42:44,537][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:42:44,862][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:42:45,187][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:42:45,512][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:42:45,840][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:42:46,167][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:42:46,494][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:42:46,821][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:42:47,149][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:42:47,476][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:42:47,801][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:42:48,127][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:42:48,454][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:42:48,784][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:42:49,110][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:42:49,438][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:42:49,764][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:42:50,093][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:42:50,419][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:42:50,744][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:42:51,070][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:42:51,398][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:42:51,722][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:42:52,046][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:42:52,372][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:42:52,700][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:42:53,391][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:42:54,129][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:42:54,130][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:42:54,131][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:42:55,183][__main__][INFO] - Iteration 422 took 23s (38.46% Gen, 56.99% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 37m 28s. Estimated total time: 19h 16m 35s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 45s.
[2025-11-13 10:42:55,185][__main__][INFO] - Starting iteration 422.
[2025-11-13 10:42:55,188][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:42:55,188][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:43:04,852][__main__][INFO] - Number of regex retries in iteration 422: 0
[2025-11-13 10:43:04,853][__main__][INFO] - agents played in iteration 422 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:43:05,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:05,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:05,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:05,423][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:05,424][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:43:05,424][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:43:06,205][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:43:06,502][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:43:06,827][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:43:07,151][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:43:07,477][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:43:07,802][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:43:08,132][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:43:08,452][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:43:08,779][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:43:09,104][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:43:09,435][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:43:09,757][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:43:10,082][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:43:10,407][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:43:10,737][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:43:11,059][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:43:11,385][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:43:11,712][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:43:12,041][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:43:12,365][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:43:12,691][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:43:13,019][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:43:13,344][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:43:13,668][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:43:13,993][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:43:14,324][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:43:14,650][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:43:14,980][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:43:15,310][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:43:15,642][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:43:15,966][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:43:16,292][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:43:16,621][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:43:17,333][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:43:18,058][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:43:18,060][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:43:18,061][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:43:19,052][__main__][INFO] - Iteration 423 took 23s (40.50% Gen, 55.35% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 13m 43s. Estimated total time: 19h 53m 14s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 46s, 500 more iterations: 3h 18m 52s.
[2025-11-13 10:43:19,054][__main__][INFO] - Starting iteration 423.
[2025-11-13 10:43:19,058][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:43:19,059][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:43:28,505][__main__][INFO] - Number of regex retries in iteration 423: 0
[2025-11-13 10:43:28,505][__main__][INFO] - agents played in iteration 423 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:43:28,977][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:29,010][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:29,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:29,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:29,077][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:43:29,078][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:43:29,855][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:43:30,151][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:43:30,477][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:43:30,811][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:43:31,137][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:43:31,463][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:43:31,788][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:43:32,113][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:43:32,439][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:43:32,764][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:43:33,090][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:43:33,416][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:43:33,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:43:34,065][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:43:34,390][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:43:34,716][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:43:35,041][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:43:35,372][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:43:35,696][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:43:36,023][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:43:36,349][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:43:36,675][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:43:36,999][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:43:37,327][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:43:37,652][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:43:37,977][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:43:38,302][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:43:38,628][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:43:38,954][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:43:39,279][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:43:39,605][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:43:39,931][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:43:40,253][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:43:40,945][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:43:41,676][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:43:41,677][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:43:41,679][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:43:42,834][__main__][INFO] - Iteration 424 took 23s (39.73% Gen, 55.40% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 8m 58s. Estimated total time: 19h 48m 52s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 37s, 500 more iterations: 3h 18m 8s.
[2025-11-13 10:43:42,836][__main__][INFO] - Starting iteration 424.
[2025-11-13 10:43:42,840][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:43:42,840][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:43:52,216][__main__][INFO] - Number of regex retries in iteration 424: 0
[2025-11-13 10:43:52,217][__main__][INFO] - agents played in iteration 424 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:43:52,687][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:52,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:52,754][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:52,788][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:43:52,788][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:43:52,789][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:43:53,574][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:43:53,872][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:43:54,197][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:43:54,525][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:43:54,850][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:43:55,176][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:43:55,501][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:43:55,828][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:43:56,154][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:43:56,480][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:43:56,804][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:43:57,130][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:43:57,454][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:43:57,785][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:43:58,111][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:43:58,436][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:43:58,763][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:43:59,091][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:43:59,416][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:43:59,740][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:44:00,071][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:44:00,399][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:44:00,727][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:44:01,054][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:44:01,380][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:44:01,706][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:44:02,033][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:44:02,359][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:44:02,689][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:44:03,019][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:44:03,348][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:44:03,677][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:44:04,004][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:44:04,720][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:44:05,458][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:44:05,460][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:44:05,461][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:44:06,456][__main__][INFO] - Iteration 425 took 23s (39.70% Gen, 56.08% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 35s. Estimated total time: 19h 40m 52s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 21s, 500 more iterations: 3h 16m 48s.
[2025-11-13 10:44:06,458][__main__][INFO] - Starting iteration 425.
[2025-11-13 10:44:06,462][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:44:06,462][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:44:14,899][__main__][INFO] - Number of regex retries in iteration 425: 0
[2025-11-13 10:44:14,900][__main__][INFO] - agents played in iteration 425 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:44:15,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:15,441][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:15,841][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:15,875][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:15,876][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:44:15,877][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:44:16,649][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:44:16,947][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:44:17,273][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:44:17,600][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:44:17,933][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:44:18,250][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:44:18,575][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:44:18,900][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:44:19,235][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:44:19,551][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:44:19,876][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:44:20,203][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:44:20,528][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:44:20,854][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:44:21,180][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:44:21,506][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:44:21,837][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:44:22,156][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:44:22,481][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:44:22,811][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:44:23,144][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:44:23,468][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:44:23,796][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:44:24,122][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:44:24,446][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:44:24,773][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:44:25,101][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:44:25,427][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:44:25,751][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:44:26,077][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:44:26,403][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:44:26,728][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:44:27,054][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:44:27,809][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:44:28,536][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:44:28,537][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:44:28,539][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:44:29,566][__main__][INFO] - Iteration 426 took 23s (36.52% Gen, 59.03% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 34m 36s. Estimated total time: 19h 15m 17s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 32s.
[2025-11-13 10:44:29,568][__main__][INFO] - Starting iteration 426.
[2025-11-13 10:44:29,572][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:44:29,572][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:44:38,961][__main__][INFO] - Number of regex retries in iteration 426: 0
[2025-11-13 10:44:38,961][__main__][INFO] - agents played in iteration 426 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:44:39,433][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:39,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:39,500][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:39,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:44:39,534][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:44:39,535][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:44:40,330][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:44:40,625][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:44:40,950][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:44:41,274][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:44:41,605][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:44:41,930][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:44:42,256][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:44:42,583][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:44:42,909][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:44:43,234][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:44:43,561][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:44:43,887][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:44:44,215][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:44:44,545][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:44:44,872][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:44:45,198][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:44:45,529][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:44:45,859][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:44:46,187][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:44:46,512][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:44:46,840][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:44:47,168][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:44:47,495][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:44:47,821][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:44:48,148][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:44:48,474][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:44:48,799][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:44:49,125][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:44:49,454][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:44:49,780][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:44:50,107][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:44:50,434][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:44:50,759][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:44:51,458][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:44:52,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:44:52,203][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:44:52,205][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:44:53,203][__main__][INFO] - Iteration 427 took 23s (39.73% Gen, 56.04% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 30s. Estimated total time: 19h 41m 34s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 23s, 500 more iterations: 3h 16m 55s.
[2025-11-13 10:44:53,205][__main__][INFO] - Starting iteration 427.
[2025-11-13 10:44:53,212][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:44:53,213][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:45:01,996][__main__][INFO] - Number of regex retries in iteration 427: 0
[2025-11-13 10:45:01,996][__main__][INFO] - agents played in iteration 427 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:45:02,479][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:02,512][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:02,545][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:02,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:02,580][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:45:02,580][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:45:03,361][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:45:03,657][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:45:03,991][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:45:04,316][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:45:04,641][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:45:04,967][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:45:05,295][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:45:05,621][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:45:05,946][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:45:06,272][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:45:06,601][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:45:06,926][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:45:07,252][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:45:07,578][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:45:07,905][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:45:08,231][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:45:08,558][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:45:08,883][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:45:09,212][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:45:09,542][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:45:09,869][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:45:10,198][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:45:10,524][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:45:10,850][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:45:11,177][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:45:11,502][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:45:11,828][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:45:12,158][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:45:12,483][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:45:12,808][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:45:13,135][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:45:13,461][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:45:13,788][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:45:14,512][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:45:15,242][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:45:15,243][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:45:15,245][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:45:16,485][__main__][INFO] - Iteration 428 took 23s (37.73% Gen, 56.92% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 42m 22s. Estimated total time: 19h 23m 49s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 47s, 500 more iterations: 3h 13m 58s.
[2025-11-13 10:45:16,487][__main__][INFO] - Starting iteration 428.
[2025-11-13 10:45:16,490][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:45:16,490][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:45:25,095][__main__][INFO] - Number of regex retries in iteration 428: 0
[2025-11-13 10:45:25,096][__main__][INFO] - agents played in iteration 428 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:45:25,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:25,608][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:25,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:25,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:25,675][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:45:25,676][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:45:26,478][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:45:26,773][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:45:27,099][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:45:27,428][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:45:27,753][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:45:28,078][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:45:28,405][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:45:28,738][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:45:29,056][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:45:29,381][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:45:29,707][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:45:30,038][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:45:30,358][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:45:30,683][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:45:31,009][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:45:31,333][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:45:31,658][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:45:31,984][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:45:32,309][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:45:32,634][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:45:32,960][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:45:33,286][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:45:33,612][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:45:33,939][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:45:34,265][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:45:34,592][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:45:34,919][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:45:35,246][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:45:35,577][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:45:35,907][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:45:36,234][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:45:36,564][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:45:36,891][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:45:37,634][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:45:38,403][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:45:38,404][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:45:38,406][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:45:39,594][__main__][INFO] - Iteration 429 took 23s (37.24% Gen, 57.61% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 33m 24s. Estimated total time: 19h 15m 15s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 32s.
[2025-11-13 10:45:39,596][__main__][INFO] - Starting iteration 429.
[2025-11-13 10:45:39,599][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:45:39,600][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:45:48,591][__main__][INFO] - Number of regex retries in iteration 429: 0
[2025-11-13 10:45:48,592][__main__][INFO] - agents played in iteration 429 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:45:49,063][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:49,096][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:49,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:49,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:45:49,164][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:45:49,165][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:45:49,961][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:45:50,256][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:45:50,581][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:45:50,906][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:45:51,239][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:45:51,564][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:45:51,889][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:45:52,215][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:45:52,545][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:45:52,870][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:45:53,197][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:45:53,522][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:45:53,847][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:45:54,171][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:45:54,497][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:45:54,822][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:45:55,148][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:45:55,474][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:45:55,799][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:45:56,126][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:45:56,452][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:45:56,777][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:45:57,103][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:45:57,428][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:45:57,756][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:45:58,081][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:45:58,405][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:45:58,731][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:45:59,056][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:45:59,382][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:45:59,710][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:46:00,036][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:46:00,365][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:46:01,145][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:46:01,995][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:46:01,996][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:46:01,998][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:46:03,069][__main__][INFO] - Iteration 430 took 23s (38.31% Gen, 57.12% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 51m 17s. Estimated total time: 19h 33m 31s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 7s, 500 more iterations: 3h 15m 35s.
[2025-11-13 10:46:03,071][__main__][INFO] - Starting iteration 430.
[2025-11-13 10:46:03,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 42 and human policies 1.
[2025-11-13 10:46:03,075][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:46:12,496][__main__][INFO] - Number of regex retries in iteration 430: 0
[2025-11-13 10:46:12,496][__main__][INFO] - agents played in iteration 430 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:46:12,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:12,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:13,027][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:13,061][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:13,062][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:46:13,062][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:46:13,864][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:46:14,160][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:46:14,488][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:46:14,812][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:46:15,138][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:46:15,463][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:46:15,789][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:46:16,115][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:46:16,440][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:46:16,765][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:46:17,091][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:46:17,417][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:46:17,742][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:46:18,067][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:46:18,393][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:46:18,719][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:46:19,044][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:46:19,369][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:46:19,694][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:46:20,019][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:46:20,345][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:46:20,676][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:46:21,004][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:46:21,335][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:46:21,661][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:46:21,985][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:46:22,310][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:46:22,634][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:46:22,959][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:46:23,284][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:46:23,610][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:46:23,940][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:46:24,267][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:46:25,013][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:46:25,772][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:46:25,774][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:46:25,775][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:46:27,822][__main__][INFO] - Iteration 431 took 24s (38.07% Gen, 53.66% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 54m 48s. Estimated total time: 20h 37m 27s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 14s, 500 more iterations: 3h 26m 14s.
[2025-11-13 10:46:27,824][__main__][INFO] - Starting iteration 431.
[2025-11-13 10:46:27,828][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:46:27,828][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:46:37,443][__main__][INFO] - Number of regex retries in iteration 431: 0
[2025-11-13 10:46:37,444][__main__][INFO] - agents played in iteration 431 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:46:37,916][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:37,949][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:37,983][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:38,016][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:46:38,017][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:46:38,018][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:46:38,818][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:46:39,115][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:46:39,441][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:46:39,769][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:46:40,094][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:46:40,419][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:46:40,744][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:46:41,069][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:46:41,394][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:46:41,720][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:46:42,046][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:46:42,370][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:46:42,696][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:46:43,021][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:46:43,346][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:46:43,676][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:46:43,999][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:46:44,324][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:46:44,648][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:46:44,974][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:46:45,299][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:46:45,624][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:46:45,949][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:46:46,273][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:46:46,602][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:46:46,928][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:46:47,258][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:46:47,589][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:46:47,920][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:46:48,247][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:46:48,573][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:46:48,898][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:46:49,224][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:46:49,934][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:46:50,680][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:46:50,681][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:46:50,683][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:46:51,696][__main__][INFO] - Iteration 432 took 23s (40.28% Gen, 55.47% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 10m 23s. Estimated total time: 19h 53m 26s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 46s, 500 more iterations: 3h 18m 54s.
[2025-11-13 10:46:51,698][__main__][INFO] - Starting iteration 432.
[2025-11-13 10:46:51,701][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:46:51,702][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:47:00,975][__main__][INFO] - Number of regex retries in iteration 432: 0
[2025-11-13 10:47:00,976][__main__][INFO] - agents played in iteration 432 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:47:01,435][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:01,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:01,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:01,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:01,541][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:47:01,541][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:47:02,324][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:47:02,697][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:47:03,004][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:47:03,328][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:47:03,653][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:47:03,979][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:47:04,304][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:47:04,630][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:47:04,955][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:47:05,281][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:47:05,606][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:47:05,932][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:47:06,257][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:47:06,582][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:47:06,907][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:47:07,232][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:47:07,562][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:47:07,886][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:47:08,212][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:47:08,536][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:47:08,864][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:47:09,190][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:47:09,515][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:47:09,839][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:47:10,167][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:47:10,492][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:47:10,816][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:47:11,143][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:47:11,470][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:47:11,798][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:47:12,124][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:47:12,455][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:47:12,775][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:47:13,497][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:47:14,260][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:47:14,261][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:47:14,262][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:47:15,303][__main__][INFO] - Iteration 433 took 23s (39.29% Gen, 56.29% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 41s. Estimated total time: 19h 40m 7s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 20s, 500 more iterations: 3h 16m 41s.
[2025-11-13 10:47:15,305][__main__][INFO] - Starting iteration 433.
[2025-11-13 10:47:15,308][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:47:15,309][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:47:24,709][__main__][INFO] - Number of regex retries in iteration 433: 0
[2025-11-13 10:47:24,710][__main__][INFO] - agents played in iteration 433 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:47:25,176][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:25,578][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:25,613][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:25,650][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:25,652][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:47:25,653][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:47:26,436][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:47:26,733][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:47:27,057][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:47:27,383][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:47:27,708][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:47:28,033][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:47:28,357][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:47:28,683][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:47:29,008][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:47:29,333][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:47:29,659][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:47:29,984][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:47:30,309][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:47:30,633][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:47:30,958][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:47:31,282][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:47:31,608][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:47:31,933][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:47:32,260][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:47:32,586][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:47:32,915][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:47:33,241][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:47:33,574][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:47:33,892][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:47:34,218][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:47:34,543][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:47:34,875][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:47:35,197][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:47:35,526][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:47:35,856][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:47:36,191][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:47:36,515][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:47:36,842][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:47:37,589][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:47:38,316][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:47:38,318][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:47:38,319][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:47:39,539][__main__][INFO] - Iteration 434 took 24s (38.80% Gen, 56.16% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 27m 43s. Estimated total time: 20h 11m 34s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 23s, 500 more iterations: 3h 21m 55s.
[2025-11-13 10:47:39,541][__main__][INFO] - Starting iteration 434.
[2025-11-13 10:47:39,544][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:47:39,545][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:47:48,807][__main__][INFO] - Number of regex retries in iteration 434: 0
[2025-11-13 10:47:48,808][__main__][INFO] - agents played in iteration 434 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:47:49,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:49,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:49,356][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:49,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:47:49,390][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:47:49,390][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:47:50,152][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:47:50,449][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:47:50,775][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:47:51,101][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:47:51,427][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:47:51,752][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:47:52,078][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:47:52,403][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:47:52,727][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:47:53,053][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:47:53,380][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:47:53,704][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:47:54,029][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:47:54,353][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:47:54,679][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:47:55,004][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:47:55,329][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:47:55,654][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:47:55,979][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:47:56,304][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:47:56,630][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:47:56,955][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:47:57,279][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:47:57,605][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:47:57,930][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:47:58,256][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:47:58,580][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:47:58,907][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:47:59,234][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:47:59,562][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:47:59,887][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:48:00,217][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:48:00,544][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:48:01,279][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:48:02,003][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:48:02,005][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:48:02,006][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:48:03,013][__main__][INFO] - Iteration 435 took 23s (39.47% Gen, 56.24% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 49m 15s. Estimated total time: 19h 33m 29s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 6s, 500 more iterations: 3h 15m 34s.
[2025-11-13 10:48:03,015][__main__][INFO] - Starting iteration 435.
[2025-11-13 10:48:03,019][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:48:03,019][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:48:12,943][__main__][INFO] - Number of regex retries in iteration 435: 0
[2025-11-13 10:48:12,943][__main__][INFO] - agents played in iteration 435 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:48:13,420][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:13,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:13,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:13,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:13,523][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:48:13,524][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:48:14,315][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:48:14,611][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:48:14,937][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:48:15,264][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:48:15,595][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:48:15,917][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:48:16,242][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:48:16,567][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:48:16,895][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:48:17,217][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:48:17,541][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:48:17,866][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:48:18,195][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:48:18,519][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:48:18,844][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:48:19,168][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:48:19,496][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:48:19,821][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:48:20,146][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:48:20,470][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:48:20,799][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:48:21,122][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:48:21,447][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:48:21,774][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:48:22,099][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:48:22,423][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:48:22,751][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:48:23,079][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:48:23,406][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:48:23,731][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:48:24,058][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:48:24,384][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:48:24,709][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:48:25,424][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:48:26,175][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:48:26,177][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:48:26,178][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:48:27,350][__main__][INFO] - Iteration 436 took 24s (40.79% Gen, 54.39% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 31m 57s. Estimated total time: 20h 16m 35s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 33s, 500 more iterations: 3h 22m 45s.
[2025-11-13 10:48:27,352][__main__][INFO] - Starting iteration 436.
[2025-11-13 10:48:27,355][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:48:27,356][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:48:36,619][__main__][INFO] - Number of regex retries in iteration 436: 0
[2025-11-13 10:48:36,619][__main__][INFO] - agents played in iteration 436 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:48:37,091][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:37,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:37,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:37,198][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:48:37,198][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:48:37,198][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:48:38,011][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:48:38,307][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:48:38,633][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:48:38,957][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:48:39,291][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:48:39,607][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:48:39,933][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:48:40,259][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:48:40,583][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:48:40,908][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:48:41,232][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:48:41,558][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:48:41,890][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:48:42,208][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:48:42,533][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:48:42,858][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:48:43,184][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:48:43,511][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:48:43,837][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:48:44,162][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:48:44,493][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:48:44,811][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:48:45,135][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:48:45,460][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:48:45,794][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:48:46,112][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:48:46,437][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:48:46,763][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:48:47,097][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:48:47,415][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:48:47,742][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:48:48,070][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:48:48,400][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:48:49,100][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:48:49,833][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:48:49,835][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:48:49,836][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:48:50,839][__main__][INFO] - Iteration 437 took 23s (39.44% Gen, 56.28% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 49m 12s. Estimated total time: 19h 34m 14s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 8s, 500 more iterations: 3h 15m 42s.
[2025-11-13 10:48:50,841][__main__][INFO] - Starting iteration 437.
[2025-11-13 10:48:50,845][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:48:50,846][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:49:00,289][__main__][INFO] - Number of regex retries in iteration 437: 0
[2025-11-13 10:49:00,290][__main__][INFO] - agents played in iteration 437 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:49:00,759][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:00,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:00,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:00,861][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:00,861][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:49:00,862][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:49:01,654][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:49:01,950][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:49:02,276][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:49:02,608][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:49:02,927][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:49:03,252][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:49:03,577][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:49:03,904][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:49:04,230][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:49:04,555][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:49:04,881][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:49:05,211][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:49:05,532][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:49:05,857][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:49:06,182][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:49:06,514][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:49:06,836][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:49:07,162][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:49:07,490][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:49:07,818][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:49:08,140][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:49:08,469][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:49:08,796][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:49:09,122][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:49:09,449][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:49:09,774][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:49:10,101][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:49:10,428][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:49:10,752][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:49:11,077][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:49:11,405][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:49:11,730][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:49:12,055][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:49:12,752][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:49:13,508][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:49:13,510][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:49:13,511][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:49:14,509][__main__][INFO] - Iteration 438 took 23s (39.90% Gen, 55.87% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 57m 48s. Estimated total time: 19h 43m 13s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 26s, 500 more iterations: 3h 17m 12s.
[2025-11-13 10:49:14,511][__main__][INFO] - Starting iteration 438.
[2025-11-13 10:49:14,514][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:49:14,514][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:49:23,750][__main__][INFO] - Number of regex retries in iteration 438: 0
[2025-11-13 10:49:23,751][__main__][INFO] - agents played in iteration 438 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:49:24,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:24,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:24,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:24,321][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:24,322][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:49:24,323][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:49:25,104][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:49:25,399][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:49:25,724][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:49:26,050][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:49:26,378][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:49:26,704][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:49:27,028][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:49:27,354][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:49:27,679][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:49:28,004][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:49:28,328][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:49:28,654][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:49:28,979][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:49:29,304][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:49:29,629][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:49:29,956][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:49:30,281][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:49:30,607][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:49:30,933][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:49:31,259][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:49:31,588][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:49:31,914][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:49:32,239][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:49:32,566][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:49:32,897][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:49:33,226][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:49:33,551][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:49:33,877][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:49:34,205][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:49:34,532][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:49:34,858][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:49:35,182][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:49:35,510][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:49:36,257][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:49:37,002][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:49:37,004][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:49:37,005][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:49:38,067][__main__][INFO] - Iteration 439 took 23s (39.21% Gen, 56.27% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 51s. Estimated total time: 19h 37m 41s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 15s, 500 more iterations: 3h 16m 16s.
[2025-11-13 10:49:38,069][__main__][INFO] - Starting iteration 439.
[2025-11-13 10:49:38,072][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:49:38,073][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:49:47,651][__main__][INFO] - Number of regex retries in iteration 439: 0
[2025-11-13 10:49:47,652][__main__][INFO] - agents played in iteration 439 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:49:48,127][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:48,161][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:48,194][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:48,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:49:48,229][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:49:48,229][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:49:49,015][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:49:49,310][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:49:49,635][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:49:49,959][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:49:50,285][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:49:50,609][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:49:50,932][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:49:51,257][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:49:51,584][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:49:51,908][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:49:52,233][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:49:52,559][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:49:52,884][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:49:53,209][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:49:53,536][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:49:53,861][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:49:54,190][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:49:54,515][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:49:54,844][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:49:55,171][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:49:55,507][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:49:55,837][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:49:56,167][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:49:56,496][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:49:56,824][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:49:57,152][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:49:57,477][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:49:57,801][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:49:58,127][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:49:58,450][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:49:58,779][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:49:59,105][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:49:59,430][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:50:00,138][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:50:00,882][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:50:00,883][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:50:00,885][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:50:01,908][__main__][INFO] - Iteration 440 took 23s (40.19% Gen, 55.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 5m 36s. Estimated total time: 19h 51m 49s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 38s.
[2025-11-13 10:50:01,910][__main__][INFO] - Starting iteration 440.
[2025-11-13 10:50:01,913][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 43 and human policies 1.
[2025-11-13 10:50:01,913][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:50:10,885][__main__][INFO] - Number of regex retries in iteration 440: 0
[2025-11-13 10:50:10,886][__main__][INFO] - agents played in iteration 440 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:50:11,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:11,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:11,431][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:11,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:11,467][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:50:11,467][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:50:12,263][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:50:12,565][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:50:12,890][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:50:13,216][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:50:13,541][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:50:13,865][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:50:14,190][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:50:14,514][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:50:14,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:50:15,164][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:50:15,490][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:50:15,814][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:50:16,139][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:50:16,464][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:50:16,789][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:50:17,115][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:50:17,441][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:50:17,765][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:50:18,089][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:50:18,413][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:50:18,742][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:50:19,068][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:50:19,392][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:50:19,719][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:50:20,044][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:50:20,372][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:50:20,700][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:50:21,031][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:50:21,359][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:50:21,684][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:50:22,011][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:50:22,335][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:50:22,661][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:50:23,404][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:50:24,140][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:50:24,141][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:50:24,143][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:50:26,547][__main__][INFO] - Iteration 441 took 24s (36.42% Gen, 53.81% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 45m 7s. Estimated total time: 20h 31m 45s. Time estimates for 10 more iterations: 4m 6s, 100 more iterations: 41m 3s, 500 more iterations: 3h 25m 17s.
[2025-11-13 10:50:26,549][__main__][INFO] - Starting iteration 441.
[2025-11-13 10:50:26,552][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:50:26,552][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:50:36,104][__main__][INFO] - Number of regex retries in iteration 441: 0
[2025-11-13 10:50:36,104][__main__][INFO] - agents played in iteration 441 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:50:36,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,631][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:50:36,700][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:50:36,701][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:50:37,482][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:50:37,779][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:50:38,104][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:50:38,430][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:50:38,757][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:50:39,080][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:50:39,405][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:50:39,734][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:50:40,060][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:50:40,386][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:50:40,711][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:50:41,036][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:50:41,363][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:50:41,688][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:50:42,013][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:50:42,339][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:50:42,664][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:50:42,989][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:50:43,314][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:50:43,639][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:50:43,966][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:50:44,291][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:50:44,617][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:50:44,943][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:50:45,269][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:50:45,593][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:50:45,919][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:50:46,244][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:50:46,569][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:50:46,895][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:50:47,220][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:50:47,546][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:50:47,872][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:50:48,597][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:50:49,495][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:50:49,496][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:50:49,498][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:50:50,489][__main__][INFO] - Iteration 442 took 23s (39.90% Gen, 55.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 9m 53s. Estimated total time: 19h 56m 55s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 53s, 500 more iterations: 3h 19m 29s.
[2025-11-13 10:50:50,491][__main__][INFO] - Starting iteration 442.
[2025-11-13 10:50:50,494][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:50:50,495][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:50:59,778][__main__][INFO] - Number of regex retries in iteration 442: 0
[2025-11-13 10:50:59,778][__main__][INFO] - agents played in iteration 442 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:51:00,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,314][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:00,347][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:51:00,348][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:51:01,134][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:51:01,429][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:51:01,754][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:51:02,080][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:51:02,407][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:51:02,733][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:51:03,059][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:51:03,385][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:51:03,711][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:51:04,039][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:51:04,364][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:51:04,689][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:51:05,014][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:51:05,341][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:51:05,666][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:51:05,992][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:51:06,317][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:51:06,643][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:51:06,969][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:51:07,297][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:51:07,629][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:51:07,955][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:51:08,283][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:51:08,608][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:51:08,940][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:51:09,262][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:51:09,587][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:51:09,914][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:51:10,249][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:51:10,567][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:51:10,891][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:51:11,215][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:51:11,549][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:51:12,300][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:51:13,073][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:51:13,075][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:51:13,076][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:51:14,244][__main__][INFO] - Iteration 443 took 23s (39.09% Gen, 55.99% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 6s. Estimated total time: 19h 47m 32s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 35s, 500 more iterations: 3h 17m 55s.
[2025-11-13 10:51:14,246][__main__][INFO] - Starting iteration 443.
[2025-11-13 10:51:14,250][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:51:14,250][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:51:23,201][__main__][INFO] - Number of regex retries in iteration 443: 0
[2025-11-13 10:51:23,202][__main__][INFO] - agents played in iteration 443 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:51:23,675][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:23,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:23,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:23,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:23,776][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:51:23,776][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:51:24,557][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:51:24,856][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:51:25,183][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:51:25,509][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:51:25,836][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:51:26,162][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:51:26,487][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:51:26,812][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:51:27,137][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:51:27,463][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:51:27,791][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:51:28,116][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:51:28,445][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:51:28,769][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:51:29,094][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:51:29,421][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:51:29,746][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:51:30,073][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:51:30,398][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:51:30,730][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:51:31,057][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:51:31,384][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:51:31,710][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:51:32,036][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:51:32,362][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:51:32,687][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:51:33,013][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:51:33,336][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:51:33,661][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:51:33,986][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:51:34,313][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:51:34,638][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:51:34,969][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:51:35,842][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:51:36,597][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:51:36,598][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:51:36,600][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:51:37,589][__main__][INFO] - Iteration 444 took 23s (38.35% Gen, 57.40% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 39m 12s. Estimated total time: 19h 27m 1s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 54s, 500 more iterations: 3h 14m 30s.
[2025-11-13 10:51:37,591][__main__][INFO] - Starting iteration 444.
[2025-11-13 10:51:37,595][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:51:37,596][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:51:46,680][__main__][INFO] - Number of regex retries in iteration 444: 0
[2025-11-13 10:51:46,680][__main__][INFO] - agents played in iteration 444 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:51:47,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:47,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:47,247][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:47,281][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:51:47,281][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:51:47,282][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:51:48,076][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:51:48,378][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:51:48,699][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:51:49,023][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:51:49,349][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:51:49,675][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:51:49,998][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:51:50,322][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:51:50,646][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:51:50,973][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:51:51,296][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:51:51,623][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:51:51,949][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:51:52,276][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:51:52,598][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:51:52,923][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:51:53,248][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:51:53,577][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:51:53,897][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:51:54,222][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:51:54,547][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:51:54,875][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:51:55,201][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:51:55,533][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:51:55,857][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:51:56,186][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:51:56,512][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:51:56,836][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:51:57,162][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:51:57,485][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:51:57,811][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:51:58,140][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:51:58,468][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:51:59,226][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:51:59,960][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:51:59,961][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:51:59,963][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:52:00,979][__main__][INFO] - Iteration 445 took 23s (38.84% Gen, 56.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 41m 1s. Estimated total time: 19h 29m 13s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 58s, 500 more iterations: 3h 14m 52s.
[2025-11-13 10:52:00,981][__main__][INFO] - Starting iteration 445.
[2025-11-13 10:52:00,985][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:52:00,985][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:52:10,339][__main__][INFO] - Number of regex retries in iteration 445: 0
[2025-11-13 10:52:10,339][__main__][INFO] - agents played in iteration 445 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:52:10,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:10,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:10,883][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:10,918][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:10,918][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:52:10,919][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:52:11,706][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:52:12,003][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:52:12,338][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:52:12,665][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:52:12,990][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:52:13,316][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:52:13,648][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:52:13,973][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:52:14,299][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:52:14,625][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:52:14,950][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:52:15,275][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:52:15,601][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:52:15,927][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:52:16,255][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:52:16,581][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:52:16,908][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:52:17,234][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:52:17,560][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:52:17,889][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:52:18,216][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:52:18,547][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:52:18,878][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:52:19,202][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:52:19,528][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:52:19,853][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:52:20,179][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:52:20,505][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:52:20,834][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:52:21,166][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:52:21,497][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:52:21,825][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:52:22,150][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:52:22,903][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:52:23,638][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:52:23,640][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:52:23,642][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:52:24,678][__main__][INFO] - Iteration 446 took 23s (39.48% Gen, 56.14% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 6s. Estimated total time: 19h 44m 42s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 29s, 500 more iterations: 3h 17m 27s.
[2025-11-13 10:52:24,680][__main__][INFO] - Starting iteration 446.
[2025-11-13 10:52:24,684][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:52:24,685][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:52:34,122][__main__][INFO] - Number of regex retries in iteration 446: 0
[2025-11-13 10:52:34,123][__main__][INFO] - agents played in iteration 446 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:52:34,609][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:34,643][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:34,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:34,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:34,711][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:52:34,711][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:52:35,477][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:52:35,780][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:52:36,099][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:52:36,424][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:52:36,750][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:52:37,076][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:52:37,401][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:52:37,727][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:52:38,052][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:52:38,378][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:52:38,703][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:52:39,029][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:52:39,355][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:52:39,680][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:52:40,006][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:52:40,331][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:52:40,657][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:52:40,982][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:52:41,307][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:52:41,632][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:52:41,958][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:52:42,284][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:52:42,609][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:52:42,934][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:52:43,260][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:52:43,588][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:52:43,911][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:52:44,235][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:52:44,559][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:52:44,888][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:52:45,208][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:52:45,532][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:52:45,857][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:52:46,610][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:52:47,339][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:52:47,340][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:52:47,343][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:52:48,386][__main__][INFO] - Iteration 447 took 23s (39.82% Gen, 55.78% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 9s. Estimated total time: 19h 45m 9s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 31s.
[2025-11-13 10:52:48,388][__main__][INFO] - Starting iteration 447.
[2025-11-13 10:52:48,391][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:52:48,392][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:52:57,282][__main__][INFO] - Number of regex retries in iteration 447: 0
[2025-11-13 10:52:57,282][__main__][INFO] - agents played in iteration 447 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:52:57,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:57,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:57,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:57,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:52:57,850][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:52:57,850][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:52:58,655][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:52:58,950][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:52:59,279][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:52:59,606][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:52:59,931][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:53:00,258][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:53:00,584][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:53:00,908][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:53:01,233][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:53:01,558][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:53:01,883][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:53:02,207][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:53:02,532][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:53:02,856][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:53:03,182][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:53:03,507][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:53:03,832][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:53:04,156][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:53:04,481][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:53:04,810][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:53:05,139][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:53:05,470][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:53:05,796][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:53:06,125][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:53:06,452][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:53:06,777][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:53:07,104][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:53:07,430][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:53:07,756][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:53:08,080][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:53:08,411][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:53:08,737][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:53:09,063][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:53:09,806][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:53:10,545][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:53:10,546][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:53:10,548][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:53:11,561][__main__][INFO] - Iteration 448 took 23s (38.37% Gen, 57.26% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 29m 8s. Estimated total time: 19h 18m 31s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 5s.
[2025-11-13 10:53:11,563][__main__][INFO] - Starting iteration 448.
[2025-11-13 10:53:11,566][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:53:11,567][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:53:20,925][__main__][INFO] - Number of regex retries in iteration 448: 0
[2025-11-13 10:53:20,926][__main__][INFO] - agents played in iteration 448 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:53:21,399][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:21,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:21,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:21,499][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:21,500][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:53:21,500][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:53:22,290][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:53:22,586][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:53:22,911][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:53:23,236][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:53:23,570][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:53:23,894][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:53:24,221][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:53:24,551][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:53:24,877][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:53:25,202][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:53:25,528][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:53:25,858][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:53:26,183][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:53:26,509][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:53:26,835][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:53:27,162][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:53:27,486][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:53:27,813][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:53:28,141][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:53:28,469][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:53:28,797][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:53:29,122][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:53:29,450][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:53:29,777][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:53:30,104][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:53:30,430][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:53:30,757][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:53:31,087][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:53:31,413][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:53:31,743][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:53:32,075][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:53:32,400][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:53:32,727][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:53:33,427][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:53:34,181][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:53:34,182][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:53:34,184][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:53:35,215][__main__][INFO] - Iteration 449 took 23s (39.57% Gen, 56.06% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 52m 44s. Estimated total time: 19h 42m 30s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 25s, 500 more iterations: 3h 17m 5s.
[2025-11-13 10:53:35,218][__main__][INFO] - Starting iteration 449.
[2025-11-13 10:53:35,221][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:53:35,222][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:53:44,653][__main__][INFO] - Number of regex retries in iteration 449: 0
[2025-11-13 10:53:44,654][__main__][INFO] - agents played in iteration 449 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:53:45,112][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:45,145][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:45,179][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:45,212][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:53:45,213][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:53:45,214][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:53:46,018][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:53:46,314][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:53:46,640][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:53:46,966][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:53:47,292][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:53:47,618][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:53:47,944][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:53:48,273][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:53:48,593][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:53:48,918][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:53:49,243][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:53:49,572][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:53:49,893][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:53:50,218][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:53:50,544][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:53:50,876][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:53:51,194][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:53:51,520][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:53:51,847][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:53:52,180][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:53:52,499][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:53:52,823][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:53:53,147][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:53:53,478][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:53:53,796][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:53:54,121][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:53:54,447][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:53:54,778][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:53:55,097][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:53:55,423][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:53:55,748][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:53:56,072][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:53:56,397][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:53:57,097][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:53:57,842][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:53:57,844][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:53:57,845][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:53:58,993][__main__][INFO] - Iteration 450 took 23s (39.68% Gen, 55.49% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 58m 27s. Estimated total time: 19h 48m 37s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 37s, 500 more iterations: 3h 18m 6s.
[2025-11-13 10:53:58,995][__main__][INFO] - Starting iteration 450.
[2025-11-13 10:53:58,998][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 44 and human policies 1.
[2025-11-13 10:53:58,999][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:54:08,246][__main__][INFO] - Number of regex retries in iteration 450: 0
[2025-11-13 10:54:08,246][__main__][INFO] - agents played in iteration 450 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:54:08,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:08,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:08,787][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:08,820][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:08,821][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:54:08,822][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:54:09,603][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:54:09,898][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:54:10,224][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:54:10,549][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:54:10,880][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:54:11,205][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:54:11,534][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:54:11,859][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:54:12,186][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:54:12,511][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:54:12,837][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:54:13,162][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:54:13,488][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:54:13,813][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:54:14,138][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:54:14,463][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:54:14,789][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:54:15,114][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:54:15,441][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:54:15,766][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:54:16,095][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:54:16,420][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:54:16,744][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:54:17,069][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:54:17,399][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:54:17,725][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:54:18,052][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:54:18,378][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:54:18,704][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:54:19,029][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:54:19,355][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:54:19,682][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:54:20,008][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:54:20,710][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:54:21,457][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:54:21,458][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:54:21,460][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:54:23,420][__main__][INFO] - Iteration 451 took 24s (37.86% Gen, 54.10% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 30m 33s. Estimated total time: 20h 21m 8s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 42s, 500 more iterations: 3h 23m 31s.
[2025-11-13 10:54:23,422][__main__][INFO] - Starting iteration 451.
[2025-11-13 10:54:23,426][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:54:23,426][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:54:32,705][__main__][INFO] - Number of regex retries in iteration 451: 0
[2025-11-13 10:54:32,706][__main__][INFO] - agents played in iteration 451 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:54:33,177][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:33,211][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:33,244][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:33,278][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:33,279][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:54:33,279][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:54:34,082][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:54:34,379][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:54:34,705][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:54:35,030][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:54:35,355][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:54:35,681][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:54:36,007][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:54:36,332][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:54:36,659][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:54:36,987][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:54:37,314][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:54:37,638][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:54:37,962][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:54:38,288][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:54:38,612][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:54:38,941][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:54:39,266][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:54:39,591][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:54:39,918][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:54:40,245][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:54:40,575][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:54:40,899][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:54:41,226][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:54:41,553][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:54:41,877][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:54:42,204][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:54:42,527][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:54:42,852][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:54:43,177][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:54:43,501][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:54:43,826][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:54:44,152][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:54:44,476][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:54:45,175][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:54:45,913][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:54:45,914][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:54:45,916][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:54:46,952][__main__][INFO] - Iteration 452 took 23s (39.44% Gen, 56.15% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 45m 21s. Estimated total time: 19h 36m 20s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 3s.
[2025-11-13 10:54:46,954][__main__][INFO] - Starting iteration 452.
[2025-11-13 10:54:46,957][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:54:46,958][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:54:57,141][__main__][INFO] - Number of regex retries in iteration 452: 0
[2025-11-13 10:54:57,143][__main__][INFO] - agents played in iteration 452 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:54:57,604][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:57,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:57,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:57,704][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:54:57,705][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:54:57,705][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:54:58,492][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:54:58,788][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:54:59,117][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:54:59,440][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:54:59,766][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:55:00,091][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:55:00,418][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:55:00,746][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:55:01,070][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:55:01,393][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:55:01,718][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:55:02,041][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:55:02,365][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:55:02,689][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:55:03,014][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:55:03,340][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:55:03,668][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:55:03,993][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:55:04,322][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:55:04,649][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:55:04,975][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:55:05,299][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:55:05,623][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:55:05,948][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:55:06,271][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:55:06,596][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:55:06,920][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:55:07,245][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:55:07,569][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:55:07,893][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:55:08,218][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:55:08,542][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:55:08,866][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:55:09,576][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:55:10,415][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:55:10,416][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:55:10,418][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:55:11,459][__main__][INFO] - Iteration 453 took 24s (41.57% Gen, 54.18% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 33m 44s. Estimated total time: 20h 25m 7s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 50s, 500 more iterations: 3h 24m 11s.
[2025-11-13 10:55:11,461][__main__][INFO] - Starting iteration 453.
[2025-11-13 10:55:11,464][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:55:11,465][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:55:21,330][__main__][INFO] - Number of regex retries in iteration 453: 0
[2025-11-13 10:55:21,330][__main__][INFO] - agents played in iteration 453 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:55:21,804][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:21,839][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:21,873][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:21,907][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:21,907][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:55:21,908][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:55:22,663][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:55:22,959][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:55:23,285][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:55:23,609][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:55:23,934][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:55:24,260][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:55:24,586][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:55:24,912][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:55:25,239][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:55:25,568][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:55:25,904][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:55:26,227][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:55:26,550][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:55:26,874][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:55:27,207][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:55:27,534][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:55:27,860][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:55:28,186][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:55:28,511][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:55:28,838][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:55:29,163][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:55:29,487][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:55:29,813][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:55:30,137][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:55:30,460][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:55:30,785][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:55:31,114][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:55:31,438][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:55:31,762][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:55:32,088][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:55:32,418][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:55:32,741][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:55:33,065][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:55:33,773][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:55:34,520][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:55:34,521][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:55:34,523][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:55:35,683][__main__][INFO] - Iteration 454 took 24s (40.74% Gen, 54.47% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 19m 12s. Estimated total time: 20h 10m 59s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 21s, 500 more iterations: 3h 21m 49s.
[2025-11-13 10:55:35,685][__main__][INFO] - Starting iteration 454.
[2025-11-13 10:55:35,689][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:55:35,689][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:55:45,418][__main__][INFO] - Number of regex retries in iteration 454: 0
[2025-11-13 10:55:45,418][__main__][INFO] - agents played in iteration 454 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:55:45,879][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:45,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:45,945][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:45,979][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:55:45,979][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:55:45,980][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:55:46,754][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:55:47,051][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:55:47,377][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:55:47,703][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:55:48,030][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:55:48,355][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:55:48,684][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:55:49,010][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:55:49,334][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:55:49,659][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:55:49,984][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:55:50,310][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:55:50,634][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:55:50,961][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:55:51,290][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:55:51,615][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:55:51,940][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:55:52,267][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:55:52,592][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:55:52,917][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:55:53,242][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:55:53,566][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:55:53,890][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:55:54,216][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:55:54,542][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:55:54,866][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:55:55,190][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:55:55,516][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:55:55,841][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:55:56,165][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:55:56,489][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:55:56,814][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:55:57,139][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:55:57,837][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:55:58,588][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:55:58,590][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:55:58,591][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:55:59,581][__main__][INFO] - Iteration 455 took 23s (40.72% Gen, 55.13% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 2m 29s. Estimated total time: 19h 54m 40s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 49s, 500 more iterations: 3h 19m 6s.
[2025-11-13 10:55:59,583][__main__][INFO] - Starting iteration 455.
[2025-11-13 10:55:59,586][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:55:59,587][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:56:09,318][__main__][INFO] - Number of regex retries in iteration 455: 0
[2025-11-13 10:56:09,318][__main__][INFO] - agents played in iteration 455 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:56:09,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:09,816][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:09,850][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:09,884][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:09,885][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:56:09,885][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:56:10,675][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:56:10,972][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:56:11,298][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:56:11,631][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:56:11,951][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:56:12,278][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:56:12,608][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:56:12,936][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:56:13,262][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:56:13,588][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:56:13,914][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:56:14,238][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:56:14,565][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:56:14,891][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:56:15,215][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:56:15,541][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:56:15,871][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:56:16,195][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:56:16,522][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:56:16,848][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:56:17,175][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:56:17,501][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:56:17,825][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:56:18,149][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:56:18,473][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:56:18,798][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:56:19,122][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:56:19,450][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:56:19,772][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:56:20,096][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:56:20,422][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:56:20,750][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:56:21,072][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:56:21,785][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:56:22,541][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:56:22,542][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:56:22,544][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:56:23,526][__main__][INFO] - Iteration 456 took 23s (40.65% Gen, 55.24% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 4m 27s. Estimated total time: 19h 57m 2s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 54s, 500 more iterations: 3h 19m 30s.
[2025-11-13 10:56:23,528][__main__][INFO] - Starting iteration 456.
[2025-11-13 10:56:23,532][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:56:23,532][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:56:33,225][__main__][INFO] - Number of regex retries in iteration 456: 0
[2025-11-13 10:56:33,226][__main__][INFO] - agents played in iteration 456 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:56:33,695][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:33,729][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:33,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:33,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:33,801][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:56:33,802][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:56:34,600][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:56:34,895][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:56:35,221][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:56:35,547][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:56:35,872][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:56:36,199][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:56:36,526][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:56:36,852][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:56:37,176][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:56:37,501][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:56:37,827][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:56:38,152][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:56:38,476][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:56:38,800][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:56:39,124][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:56:39,450][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:56:39,777][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:56:40,103][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:56:40,427][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:56:40,753][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:56:41,079][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:56:41,405][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:56:41,732][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:56:42,057][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:56:42,384][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:56:42,709][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:56:43,040][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:56:43,359][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:56:43,686][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:56:44,013][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:56:44,346][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:56:44,669][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:56:44,997][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:56:45,732][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:56:46,473][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:56:46,475][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:56:46,477][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:56:47,759][__main__][INFO] - Iteration 457 took 24s (40.01% Gen, 54.69% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 18m 24s. Estimated total time: 20h 11m 23s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 22s, 500 more iterations: 3h 21m 53s.
[2025-11-13 10:56:47,761][__main__][INFO] - Starting iteration 457.
[2025-11-13 10:56:47,765][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:56:47,765][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:56:57,840][__main__][INFO] - Number of regex retries in iteration 457: 0
[2025-11-13 10:56:57,841][__main__][INFO] - agents played in iteration 457 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:56:58,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:58,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:58,369][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:58,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:56:58,404][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:56:58,405][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:56:59,210][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:56:59,507][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:56:59,833][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:57:00,161][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:57:00,483][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:57:00,813][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:57:01,141][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:57:01,466][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:57:01,792][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:57:02,118][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:57:02,446][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:57:02,770][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:57:03,097][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:57:03,421][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:57:03,747][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:57:04,072][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:57:04,407][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:57:04,735][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:57:05,061][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:57:05,385][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:57:05,714][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:57:06,038][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:57:06,363][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:57:06,689][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:57:07,025][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:57:07,348][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:57:07,672][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:57:07,997][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:57:08,333][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:57:08,658][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:57:08,985][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:57:09,309][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:57:09,633][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:57:10,337][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:57:11,076][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:57:11,077][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:57:11,079][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:57:12,106][__main__][INFO] - Iteration 458 took 24s (41.39% Gen, 54.38% Train). Generation: 10s, Training: 13s. Estimated remaining time: 17h 23m 42s. Estimated total time: 20h 17m 5s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 34s, 500 more iterations: 3h 22m 50s.
[2025-11-13 10:57:12,108][__main__][INFO] - Starting iteration 458.
[2025-11-13 10:57:12,112][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:57:12,112][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:57:21,967][__main__][INFO] - Number of regex retries in iteration 458: 0
[2025-11-13 10:57:21,968][__main__][INFO] - agents played in iteration 458 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:57:22,438][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:22,472][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:22,505][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:22,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:22,539][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:57:22,539][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:57:23,300][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:57:23,596][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:57:23,930][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:57:24,259][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:57:24,586][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:57:24,912][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:57:25,242][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:57:25,569][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:57:25,902][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:57:26,232][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:57:26,559][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:57:26,884][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:57:27,212][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:57:27,538][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:57:27,863][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:57:28,189][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:57:28,513][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:57:28,836][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:57:29,163][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:57:29,488][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:57:29,813][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:57:30,139][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:57:30,466][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:57:30,791][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:57:31,118][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:57:31,445][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:57:31,771][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:57:32,095][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:57:32,420][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:57:32,744][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:57:33,068][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:57:33,392][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:57:33,717][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:57:34,433][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:57:35,178][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:57:35,180][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:57:35,182][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:57:36,289][__main__][INFO] - Iteration 459 took 24s (40.76% Gen, 54.65% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 15m 7s. Estimated total time: 20h 8m 54s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 17s, 500 more iterations: 3h 21m 29s.
[2025-11-13 10:57:36,291][__main__][INFO] - Starting iteration 459.
[2025-11-13 10:57:36,294][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:57:36,295][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:57:46,006][__main__][INFO] - Number of regex retries in iteration 459: 0
[2025-11-13 10:57:46,006][__main__][INFO] - agents played in iteration 459 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:57:46,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:46,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:46,541][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:46,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:57:46,575][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:57:46,575][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:57:47,344][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:57:47,641][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:57:47,967][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:57:48,292][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:57:48,617][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:57:48,944][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:57:49,274][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:57:49,600][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:57:49,925][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:57:50,255][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:57:50,581][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:57:50,907][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:57:51,233][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:57:51,558][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:57:51,888][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:57:52,217][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:57:52,542][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:57:52,868][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:57:53,193][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:57:53,518][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:57:53,843][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:57:54,171][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:57:54,494][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:57:54,820][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:57:55,143][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:57:55,468][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:57:55,792][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:57:56,117][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:57:56,442][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:57:56,766][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:57:57,091][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:57:57,416][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:57:57,739][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:57:58,455][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:57:59,201][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:57:59,203][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:57:59,204][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:58:00,196][__main__][INFO] - Iteration 460 took 23s (40.63% Gen, 55.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 0m 56s. Estimated total time: 19h 55m 7s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 50s, 500 more iterations: 3h 19m 11s.
[2025-11-13 10:58:00,198][__main__][INFO] - Starting iteration 460.
[2025-11-13 10:58:00,201][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 45 and human policies 1.
[2025-11-13 10:58:00,201][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:58:09,963][__main__][INFO] - Number of regex retries in iteration 460: 0
[2025-11-13 10:58:09,964][__main__][INFO] - agents played in iteration 460 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:58:10,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:10,454][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:10,487][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:10,521][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:10,522][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:58:10,522][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:58:11,297][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:58:11,592][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:58:11,918][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:58:12,244][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:58:12,568][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:58:12,897][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:58:13,222][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:58:13,547][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:58:13,872][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:58:14,197][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:58:14,523][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:58:14,853][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:58:15,178][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:58:15,507][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:58:15,835][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:58:16,159][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:58:16,484][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:58:16,814][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:58:17,139][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:58:17,466][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:58:17,791][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:58:18,116][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:58:18,441][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:58:18,766][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:58:19,091][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:58:19,415][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:58:19,740][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:58:20,064][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:58:20,389][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:58:20,716][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:58:21,042][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:58:21,366][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:58:21,693][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:58:22,396][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:58:23,148][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:58:23,149][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:58:23,151][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:58:25,327][__main__][INFO] - Iteration 461 took 25s (38.85% Gen, 52.48% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 1m 43s. Estimated total time: 20h 56m 20s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 52s, 500 more iterations: 3h 29m 23s.
[2025-11-13 10:58:25,329][__main__][INFO] - Starting iteration 461.
[2025-11-13 10:58:25,333][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:58:25,333][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:58:34,353][__main__][INFO] - Number of regex retries in iteration 461: 0
[2025-11-13 10:58:34,354][__main__][INFO] - agents played in iteration 461 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:58:34,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:35,235][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:35,269][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:35,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:35,304][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:58:35,305][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:58:36,080][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:58:36,375][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:58:36,701][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:58:37,027][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:58:37,352][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:58:37,679][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:58:38,008][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:58:38,334][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:58:38,661][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:58:38,988][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:58:39,318][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:58:39,646][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:58:39,973][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:58:40,304][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:58:40,629][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:58:40,954][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:58:41,280][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:58:41,603][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:58:41,927][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:58:42,250][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:58:42,578][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:58:42,904][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:58:43,231][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:58:43,556][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:58:43,882][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:58:44,207][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:58:44,531][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:58:44,856][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:58:45,180][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:58:45,505][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:58:45,829][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:58:46,154][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:58:46,479][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:58:47,184][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:58:47,924][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:58:47,925][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:58:47,927][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:58:49,152][__main__][INFO] - Iteration 462 took 23s (37.87% Gen, 56.98% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 56m 0s. Estimated total time: 19h 51m 0s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 42s, 500 more iterations: 3h 18m 30s.
[2025-11-13 10:58:49,154][__main__][INFO] - Starting iteration 462.
[2025-11-13 10:58:49,157][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:58:49,158][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:58:58,409][__main__][INFO] - Number of regex retries in iteration 462: 0
[2025-11-13 10:58:58,410][__main__][INFO] - agents played in iteration 462 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:58:58,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:58,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:58,954][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:58,987][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:58:58,988][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:58:58,988][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:59:00,131][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:59:00,426][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:59:00,750][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:59:01,075][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:59:01,401][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:59:01,730][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:59:02,055][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:59:02,381][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:59:02,705][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:59:03,031][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:59:03,357][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:59:03,683][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:59:04,010][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:59:04,335][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:59:04,661][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:59:04,989][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:59:05,315][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:59:05,640][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:59:05,964][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:59:06,291][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:59:06,617][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:59:06,941][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:59:07,268][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:59:07,593][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:59:07,917][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:59:08,241][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:59:08,566][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:59:08,890][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:59:09,217][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:59:09,540][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:59:09,864][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:59:10,188][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:59:10,513][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:59:11,212][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:59:11,951][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:59:11,953][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:59:11,955][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:59:13,156][__main__][INFO] - Iteration 463 took 23s (38.55% Gen, 56.44% Train). Generation: 9s, Training: 13s. Estimated remaining time: 17h 4m 35s. Estimated total time: 19h 59m 59s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 59s, 500 more iterations: 3h 19m 59s.
[2025-11-13 10:59:13,158][__main__][INFO] - Starting iteration 463.
[2025-11-13 10:59:13,163][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:59:13,163][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:59:22,755][__main__][INFO] - Number of regex retries in iteration 463: 0
[2025-11-13 10:59:22,755][__main__][INFO] - agents played in iteration 463 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:59:23,224][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:23,257][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:23,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:23,324][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:23,324][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:59:23,325][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:59:24,107][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:59:24,402][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:59:24,728][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:59:25,056][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:59:25,381][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:59:25,706][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:59:26,032][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:59:26,359][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:59:26,684][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:59:27,008][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:59:27,333][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:59:27,658][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:59:27,984][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:59:28,310][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:59:28,636][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:59:28,966][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:59:29,293][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:59:29,617][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:59:29,941][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:59:30,269][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:59:30,594][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:59:30,917][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:59:31,241][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:59:31,572][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:59:31,892][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:59:32,219][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:59:32,543][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:59:32,867][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:59:33,192][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:59:33,515][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:59:33,840][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:59:34,164][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:59:34,489][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:59:35,185][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:59:35,929][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:59:35,930][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:59:35,932][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:59:36,913][__main__][INFO] - Iteration 464 took 23s (40.39% Gen, 55.48% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 51m 46s. Estimated total time: 19h 47m 34s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 35s, 500 more iterations: 3h 17m 55s.
[2025-11-13 10:59:36,915][__main__][INFO] - Starting iteration 464.
[2025-11-13 10:59:36,918][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:59:36,919][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 10:59:45,252][__main__][INFO] - Number of regex retries in iteration 464: 0
[2025-11-13 10:59:45,253][__main__][INFO] - agents played in iteration 464 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 10:59:45,714][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:45,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:45,779][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:45,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 10:59:45,813][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 10:59:45,814][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 10:59:46,573][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 10:59:46,868][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 10:59:47,198][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 10:59:47,519][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 10:59:47,844][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 10:59:48,168][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 10:59:48,500][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 10:59:48,821][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 10:59:49,147][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 10:59:49,473][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 10:59:49,801][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 10:59:50,122][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 10:59:50,448][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 10:59:50,774][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 10:59:51,102][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 10:59:51,425][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 10:59:51,748][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 10:59:52,074][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 10:59:52,406][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 10:59:52,731][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 10:59:53,057][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 10:59:53,382][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 10:59:53,711][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 10:59:54,038][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 10:59:54,364][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 10:59:54,693][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 10:59:55,019][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 10:59:55,349][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 10:59:55,675][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 10:59:55,999][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 10:59:56,325][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 10:59:56,654][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 10:59:56,980][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 10:59:57,694][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 10:59:58,433][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 10:59:58,435][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 10:59:58,437][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 10:59:59,480][__main__][INFO] - Iteration 465 took 22s (36.94% Gen, 58.43% Train). Generation: 8s, Training: 13s. Estimated remaining time: 15h 51m 58s. Estimated total time: 18h 48m 9s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 36s, 500 more iterations: 3h 8m 1s.
[2025-11-13 10:59:59,482][__main__][INFO] - Starting iteration 465.
[2025-11-13 10:59:59,486][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 10:59:59,486][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:00:08,076][__main__][INFO] - Number of regex retries in iteration 465: 0
[2025-11-13 11:00:08,077][__main__][INFO] - agents played in iteration 465 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:00:08,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:08,616][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:08,649][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:08,683][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:08,683][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:00:08,684][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:00:09,453][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:00:09,750][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:00:10,075][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:00:10,400][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:00:10,725][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:00:11,049][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:00:11,374][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:00:11,700][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:00:12,024][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:00:12,348][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:00:12,673][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:00:12,998][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:00:13,322][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:00:13,649][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:00:13,983][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:00:14,310][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:00:14,636][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:00:14,961][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:00:15,291][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:00:15,615][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:00:15,939][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:00:16,265][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:00:16,590][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:00:16,915][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:00:17,242][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:00:17,573][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:00:17,902][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:00:18,232][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:00:18,559][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:00:18,895][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:00:19,212][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:00:19,536][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:00:19,862][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:00:20,583][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:00:21,323][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:00:21,324][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:00:21,326][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:00:22,429][__main__][INFO] - Iteration 466 took 22s (37.44% Gen, 57.74% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 10m 40s. Estimated total time: 19h 7m 14s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 14s, 500 more iterations: 3h 11m 12s.
[2025-11-13 11:00:22,431][__main__][INFO] - Starting iteration 466.
[2025-11-13 11:00:22,435][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:00:22,436][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:00:30,846][__main__][INFO] - Number of regex retries in iteration 466: 0
[2025-11-13 11:00:30,846][__main__][INFO] - agents played in iteration 466 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:00:31,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:31,359][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:31,392][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:31,427][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:31,427][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:00:31,427][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:00:32,244][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:00:32,539][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:00:32,864][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:00:33,189][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:00:33,515][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:00:33,840][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:00:34,165][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:00:34,490][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:00:34,818][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:00:35,142][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:00:35,467][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:00:35,793][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:00:36,118][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:00:36,442][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:00:36,768][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:00:37,093][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:00:37,418][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:00:37,744][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:00:38,070][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:00:38,397][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:00:38,727][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:00:39,054][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:00:39,382][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:00:39,708][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:00:40,032][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:00:40,356][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:00:40,682][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:00:41,010][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:00:41,334][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:00:41,659][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:00:41,985][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:00:42,311][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:00:42,637][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:00:43,374][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:00:44,143][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:00:44,144][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:00:44,146][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:00:45,165][__main__][INFO] - Iteration 467 took 22s (37.00% Gen, 58.51% Train). Generation: 8s, Training: 13s. Estimated remaining time: 15h 59m 36s. Estimated total time: 18h 56m 32s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 53s, 500 more iterations: 3h 9m 25s.
[2025-11-13 11:00:45,167][__main__][INFO] - Starting iteration 467.
[2025-11-13 11:00:45,171][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:00:45,171][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:00:49,921][mllm.models.large_language_model_local][WARNING] - Response <I A> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 11:00:53,937][__main__][INFO] - Number of regex retries in iteration 467: 1
[2025-11-13 11:00:53,937][__main__][INFO] - agents played in iteration 467 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:00:54,413][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:54,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:54,483][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:54,517][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:00:54,517][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:00:54,518][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:00:55,689][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:00:55,990][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:00:56,316][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:00:56,640][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:00:56,965][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:00:57,295][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:00:57,621][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:00:57,946][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:00:58,270][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:00:58,599][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:00:58,924][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:00:59,249][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:00:59,574][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:00:59,901][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:01:00,226][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:01:00,551][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:01:00,877][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:01:01,203][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:01:01,530][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:01:01,855][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:01:02,180][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:01:02,505][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:01:02,832][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:01:03,157][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:01:03,484][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:01:03,808][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:01:04,135][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:01:04,466][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:01:04,793][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:01:05,123][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:01:05,456][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:01:05,782][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:01:06,110][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:01:06,902][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:01:07,650][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:01:07,651][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:01:07,653][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:01:08,870][__main__][INFO] - Iteration 468 took 23s (36.99% Gen, 57.87% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 47m 40s. Estimated total time: 19h 45m 0s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 30s.
[2025-11-13 11:01:08,872][__main__][INFO] - Starting iteration 468.
[2025-11-13 11:01:08,875][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:01:08,876][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:01:17,629][__main__][INFO] - Number of regex retries in iteration 468: 0
[2025-11-13 11:01:17,630][__main__][INFO] - agents played in iteration 468 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:01:18,102][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:18,136][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:18,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:18,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:18,204][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:01:18,204][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:01:18,998][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:01:19,294][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:01:19,621][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:01:19,952][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:01:20,280][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:01:20,614][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:01:20,932][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:01:21,257][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:01:21,583][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:01:21,909][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:01:22,234][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:01:22,558][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:01:22,883][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:01:23,209][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:01:23,534][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:01:23,860][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:01:24,185][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:01:24,511][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:01:24,837][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:01:25,163][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:01:25,488][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:01:25,819][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:01:26,143][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:01:26,472][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:01:26,797][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:01:27,123][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:01:27,448][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:01:27,773][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:01:28,098][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:01:28,426][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:01:28,753][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:01:29,083][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:01:29,409][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:01:30,195][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:01:30,939][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:01:30,941][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:01:30,943][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:01:31,939][__main__][INFO] - Iteration 469 took 23s (37.95% Gen, 57.72% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 15m 32s. Estimated total time: 19h 13m 15s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 26s, 500 more iterations: 3h 12m 12s.
[2025-11-13 11:01:31,942][__main__][INFO] - Starting iteration 469.
[2025-11-13 11:01:31,945][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:01:31,946][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:01:41,395][__main__][INFO] - Number of regex retries in iteration 469: 0
[2025-11-13 11:01:41,395][__main__][INFO] - agents played in iteration 469 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:01:41,878][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:41,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:41,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:41,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:01:41,981][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:01:41,981][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:01:42,758][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:01:43,054][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:01:43,381][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:01:43,708][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:01:44,037][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:01:44,365][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:01:44,691][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:01:45,016][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:01:45,341][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:01:45,669][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:01:45,994][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:01:46,320][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:01:46,646][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:01:46,970][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:01:47,296][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:01:47,620][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:01:47,945][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:01:48,271][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:01:48,595][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:01:48,921][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:01:49,247][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:01:49,580][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:01:49,906][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:01:50,230][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:01:50,557][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:01:50,895][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:01:51,226][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:01:51,552][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:01:51,878][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:01:52,207][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:01:52,535][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:01:52,861][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:01:53,196][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:01:53,982][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:01:54,729][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:01:54,731][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:01:54,733][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:01:55,901][__main__][INFO] - Iteration 470 took 23s (39.44% Gen, 55.67% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 59m 41s. Estimated total time: 19h 57m 49s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 55s, 500 more iterations: 3h 19m 38s.
[2025-11-13 11:01:55,903][__main__][INFO] - Starting iteration 470.
[2025-11-13 11:01:55,906][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 46 and human policies 1.
[2025-11-13 11:01:55,907][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:02:04,632][__main__][INFO] - Number of regex retries in iteration 470: 0
[2025-11-13 11:02:04,633][__main__][INFO] - agents played in iteration 470 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:02:05,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:05,119][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:05,151][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:05,184][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:05,184][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:02:05,185][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:02:05,926][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:02:06,222][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:02:06,554][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:02:06,885][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:02:07,215][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:02:07,542][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:02:07,878][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:02:08,209][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:02:08,538][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:02:08,864][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:02:09,202][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:02:09,526][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:02:09,854][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:02:10,179][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:02:10,509][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:02:10,834][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:02:11,159][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:02:11,484][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:02:11,809][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:02:12,133][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:02:12,459][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:02:12,784][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:02:13,112][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:02:13,437][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:02:13,762][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:02:14,089][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:02:14,419][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:02:14,743][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:02:15,069][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:02:15,395][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:02:15,721][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:02:16,046][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:02:16,371][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:02:17,128][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:02:17,859][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:02:17,860][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:02:17,862][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:02:19,927][__main__][INFO] - Iteration 471 took 24s (36.32% Gen, 55.08% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 2m 32s. Estimated total time: 20h 1m 3s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 2s, 500 more iterations: 3h 20m 10s.
[2025-11-13 11:02:19,929][__main__][INFO] - Starting iteration 471.
[2025-11-13 11:02:19,932][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:02:19,933][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:02:29,202][__main__][INFO] - Number of regex retries in iteration 471: 0
[2025-11-13 11:02:29,203][__main__][INFO] - agents played in iteration 471 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:02:29,667][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:29,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:29,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:29,770][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:29,771][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:02:29,771][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:02:30,533][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:02:30,830][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:02:31,159][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:02:31,487][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:02:31,819][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:02:32,144][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:02:32,472][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:02:32,799][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:02:33,126][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:02:33,451][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:02:33,779][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:02:34,108][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:02:34,434][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:02:34,760][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:02:35,085][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:02:35,410][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:02:35,735][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:02:36,061][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:02:36,387][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:02:36,712][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:02:37,037][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:02:37,363][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:02:37,688][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:02:38,014][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:02:38,339][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:02:38,664][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:02:38,989][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:02:39,315][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:02:39,640][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:02:39,967][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:02:40,293][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:02:40,619][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:02:40,948][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:02:41,701][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:02:42,442][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:02:42,444][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:02:42,445][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:02:43,609][__main__][INFO] - Iteration 472 took 23s (39.15% Gen, 55.93% Train). Generation: 9s, Training: 13s. Estimated remaining time: 16h 44m 59s. Estimated total time: 19h 43m 53s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 27s, 500 more iterations: 3h 17m 18s.
[2025-11-13 11:02:43,611][__main__][INFO] - Starting iteration 472.
[2025-11-13 11:02:43,615][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:02:43,616][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:02:52,494][__main__][INFO] - Number of regex retries in iteration 472: 0
[2025-11-13 11:02:52,494][__main__][INFO] - agents played in iteration 472 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:02:52,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:52,973][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:53,006][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:53,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:02:53,040][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:02:53,040][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:02:54,107][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:02:54,404][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:02:54,731][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:02:55,057][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:02:55,383][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:02:55,713][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:02:56,044][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:02:56,368][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:02:56,695][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:02:57,025][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:02:57,357][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:02:57,682][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:02:58,008][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:02:58,334][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:02:58,660][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:02:58,986][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:02:59,312][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:02:59,638][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:02:59,963][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:03:00,288][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:03:00,613][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:03:00,938][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:03:01,263][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:03:01,589][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:03:01,914][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:03:02,240][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:03:02,566][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:03:02,898][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:03:03,223][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:03:03,549][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:03:03,874][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:03:04,212][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:03:04,537][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:03:05,298][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:03:06,036][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:03:06,038][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:03:06,039][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:03:07,405][__main__][INFO] - Iteration 473 took 23s (37.32% Gen, 56.94% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 50m 12s. Estimated total time: 19h 49m 30s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 39s, 500 more iterations: 3h 18m 15s.
[2025-11-13 11:03:07,407][__main__][INFO] - Starting iteration 473.
[2025-11-13 11:03:07,410][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:03:07,411][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:03:16,106][__main__][INFO] - Number of regex retries in iteration 473: 0
[2025-11-13 11:03:16,106][__main__][INFO] - agents played in iteration 473 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:03:16,555][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:16,590][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:16,623][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:16,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:16,657][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:03:16,657][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:03:17,365][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:03:17,659][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:03:17,983][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:03:18,307][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:03:18,632][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:03:18,957][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:03:19,285][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:03:19,610][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:03:19,936][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:03:20,262][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:03:20,588][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:03:20,914][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:03:21,244][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:03:21,572][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:03:21,897][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:03:22,224][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:03:22,550][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:03:22,876][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:03:23,202][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:03:23,530][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:03:23,854][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:03:24,179][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:03:24,505][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:03:24,830][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:03:25,156][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:03:25,483][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:03:25,808][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:03:26,133][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:03:26,460][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:03:26,785][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:03:27,112][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:03:27,437][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:03:27,762][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:03:28,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 42.04%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:03:29,272][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:03:29,333][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:03:29,334][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:03:30,371][__main__][INFO] - Iteration 474 took 22s (37.87% Gen, 57.61% Train). Generation: 8s, Training: 13s. Estimated remaining time: 16h 8m 22s. Estimated total time: 19h 8m 4s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 16s, 500 more iterations: 3h 11m 20s.
[2025-11-13 11:03:30,373][__main__][INFO] - Starting iteration 474.
[2025-11-13 11:03:30,376][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:03:30,377][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:03:39,305][__main__][INFO] - Number of regex retries in iteration 474: 0
[2025-11-13 11:03:39,305][__main__][INFO] - agents played in iteration 474 are Bob_buffer, Bob, Alice_buffer, Alice
[2025-11-13 11:03:39,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:39,790][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:39,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:39,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.78%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:03:39,857][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:03:39,858][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:03:40,567][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:03:40,864][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:03:41,187][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:03:41,513][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:03:41,839][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:03:42,165][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:03:42,493][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:03:42,818][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:03:43,144][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:03:43,475][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:03:43,801][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:03:44,127][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:03:44,452][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:03:44,778][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:03:45,112][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:03:45,438][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:03:45,763][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:03:46,091][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:03:46,426][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:03:46,753][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:06:50,929][mllm.models.large_language_model_local][INFO] - Loaded 47 past agent adapters from checkpoints directory.
[2025-11-13 11:07:10,032][mllm.models.large_language_model_local][INFO] - Initializing adapter 'agent_adapter': using existing weights from output folder '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/Qwen/Qwen2.5-7B-Instruct/adapters/agent_adapter'.
[2025-11-13 11:07:11,453][mllm.models.adapter_training_wrapper][INFO] - Adapter 'agent_adapter': loaded initial weights from '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/Qwen/Qwen2.5-7B-Instruct/adapters/agent_adapter'.
[2025-11-13 11:07:11,461][mllm.models.large_language_model_local][INFO] - Initializing adapter 'critic_adapter': using existing weights from output folder '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/Qwen/Qwen2.5-7B-Instruct/adapters/critic_adapter'.
[2025-11-13 11:07:12,714][mllm.models.adapter_training_wrapper][INFO] - Adapter 'critic_adapter': loaded initial weights from '/scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/Qwen/Qwen2.5-7B-Instruct/adapters/critic_adapter'.
[2025-11-13 11:09:20,829][mllm.training.trainer_common][INFO] - Loading trainer state from /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:09:20,831][mllm.training.trainer_common][INFO] - Loading policy optimizer state from /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:09:21,643][mllm.training.trainer_common][INFO] - Loading critic optimizer state from /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:09:21,646][__main__][INFO] - Starting iteration 474.
[2025-11-13 11:09:21,651][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:09:21,652][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:09:48,480][__main__][INFO] - Number of regex retries in iteration 474: 0
[2025-11-13 11:09:48,481][__main__][INFO] - agents played in iteration 474 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:09:48,915][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:48,955][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:48,995][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:49,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 39.04%, Block Peak % of device VRAM: 19.44%, ΔTime: 00:00:00
[2025-11-13 11:09:49,034][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:09:49,035][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:09:49,661][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:09:50,395][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:09:50,727][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:09:51,055][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:09:51,382][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:09:51,712][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:09:52,041][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:09:52,373][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:09:52,697][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:09:53,022][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:09:53,352][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:09:53,683][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:09:54,006][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:09:54,331][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:09:54,656][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:09:54,980][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:09:55,308][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:09:55,634][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:09:55,956][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:09:56,281][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:09:56,606][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:09:56,930][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:09:57,252][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:09:57,576][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:09:57,900][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:09:58,224][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:09:58,548][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:09:58,871][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:09:59,194][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:09:59,519][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:09:59,851][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:10:00,182][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:10:00,507][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:10:01,169][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.78%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.98%, ΔTime: 00:00:11
[2025-11-13 11:10:02,084][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:10:02,087][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:10:02,088][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:10:03,154][__main__][INFO] - Iteration 475 took 41s (64.64% Gen, 32.79% Train). Generation: 26s, Training: 13s. Estimated remaining time: 34h 32m 0s. Estimated total time: 34h 35m 14s. Time estimates for 10 more iterations: 6m 55s, 100 more iterations: 1h 9m 10s, 500 more iterations: 5h 45m 52s.
[2025-11-13 11:10:03,159][__main__][INFO] - Starting iteration 475.
[2025-11-13 11:10:03,162][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:10:03,162][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:10:19,805][__main__][INFO] - Number of regex retries in iteration 475: 0
[2025-11-13 11:10:19,805][__main__][INFO] - agents played in iteration 475 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:10:20,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,332][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,371][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:20,371][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:10:20,372][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:10:21,068][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:10:21,365][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:10:21,696][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:10:22,029][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:10:22,355][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:10:22,680][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:10:23,003][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:10:23,327][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:10:23,651][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:10:23,974][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:10:24,300][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:10:24,624][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:10:24,949][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:10:25,276][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:10:25,600][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:10:25,924][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:10:26,248][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:10:26,571][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:10:26,895][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:10:27,226][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:10:27,552][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:10:27,880][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:10:28,213][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:10:28,536][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:10:28,867][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:10:29,194][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:10:29,520][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:10:29,846][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:10:30,172][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:10:30,499][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:10:30,826][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:10:31,152][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:10:31,477][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:10:32,141][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:10:32,875][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:10:32,877][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:10:32,878][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:10:33,818][__main__][INFO] - Iteration 476 took 30s (54.29% Gen, 42.64% Train). Generation: 16s, Training: 13s. Estimated remaining time: 25h 29m 8s. Estimated total time: 25h 32m 52s. Time estimates for 10 more iterations: 5m 6s, 100 more iterations: 51m 5s, 500 more iterations: 4h 15m 28s.
[2025-11-13 11:10:33,820][__main__][INFO] - Starting iteration 476.
[2025-11-13 11:10:33,824][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:10:33,824][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:10:46,073][__main__][INFO] - Number of regex retries in iteration 476: 0
[2025-11-13 11:10:46,073][__main__][INFO] - agents played in iteration 476 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:10:46,494][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:46,534][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:46,573][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:46,612][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:10:46,613][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:10:46,613][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:10:47,302][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:10:47,600][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:10:47,932][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:10:48,260][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:10:48,585][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:10:48,912][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:10:49,238][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:10:49,562][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:10:49,889][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:10:50,215][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:10:50,541][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:10:50,865][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:10:51,196][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:10:51,521][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:10:51,846][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:10:52,169][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:10:52,493][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:10:52,819][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:10:53,145][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:10:53,471][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:10:53,795][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:10:54,119][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:10:54,443][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:10:54,768][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:10:55,092][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:10:55,416][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:10:55,739][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:10:56,064][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:10:56,391][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:10:56,717][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:10:57,042][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:10:57,369][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:10:57,694][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:10:58,330][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:10:59,057][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:10:59,059][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:10:59,060][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:10:59,955][__main__][INFO] - Iteration 477 took 26s (46.87% Gen, 49.70% Train). Generation: 12s, Training: 12s. Estimated remaining time: 21h 42m 25s. Estimated total time: 21h 46m 35s. Time estimates for 10 more iterations: 4m 21s, 100 more iterations: 43m 33s, 500 more iterations: 3h 37m 45s.
[2025-11-13 11:10:59,957][__main__][INFO] - Starting iteration 477.
[2025-11-13 11:10:59,960][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:10:59,961][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:11:11,871][__main__][INFO] - Number of regex retries in iteration 477: 0
[2025-11-13 11:11:11,871][__main__][INFO] - agents played in iteration 477 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:11:12,303][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:12,345][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:12,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:12,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:12,426][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:11:12,426][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:11:13,117][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:11:13,415][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:11:13,742][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:11:14,071][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:11:14,396][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:11:14,720][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:11:15,043][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:11:15,366][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:11:15,690][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:11:16,014][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:11:16,340][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:11:16,665][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:11:16,990][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:11:17,315][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:11:17,639][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:11:17,964][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:11:18,289][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:11:18,612][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:11:18,936][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:11:19,260][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:11:19,585][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:11:19,909][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:11:20,232][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:11:20,556][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:11:20,880][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:11:21,210][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:11:21,533][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:11:21,860][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:11:22,184][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:11:22,508][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:11:22,831][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:11:23,156][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:11:23,481][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:11:24,150][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:11:24,883][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:11:24,885][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:11:24,887][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:11:25,804][__main__][INFO] - Iteration 478 took 25s (46.08% Gen, 50.36% Train). Generation: 11s, Training: 13s. Estimated remaining time: 21h 27m 38s. Estimated total time: 21h 32m 15s. Time estimates for 10 more iterations: 4m 18s, 100 more iterations: 43m 4s, 500 more iterations: 3h 35m 22s.
[2025-11-13 11:11:25,807][__main__][INFO] - Starting iteration 478.
[2025-11-13 11:11:25,810][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:11:25,810][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:11:37,152][__main__][INFO] - Number of regex retries in iteration 478: 0
[2025-11-13 11:11:37,153][__main__][INFO] - agents played in iteration 478 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:11:37,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,622][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,663][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,705][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:11:37,705][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:11:37,706][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:11:38,401][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:11:38,698][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:11:39,026][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:11:39,353][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:11:39,678][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:11:40,005][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:11:40,332][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:11:40,657][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:11:40,981][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:11:41,305][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:11:41,632][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:11:41,957][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:11:42,281][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:11:42,606][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:11:42,930][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:11:43,253][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:11:43,580][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:11:43,908][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:11:44,234][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:11:44,557][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:11:44,881][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:11:45,205][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:11:45,530][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:11:45,857][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:11:46,183][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:11:46,510][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:11:46,837][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:11:47,162][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:11:47,490][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:11:47,816][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:11:48,140][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:11:48,465][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:11:48,790][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:11:49,477][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:11:50,215][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:11:50,217][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:11:50,219][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:11:51,096][__main__][INFO] - Iteration 479 took 25s (44.85% Gen, 51.67% Train). Generation: 11s, Training: 13s. Estimated remaining time: 20h 59m 18s. Estimated total time: 21h 4m 20s. Time estimates for 10 more iterations: 4m 12s, 100 more iterations: 42m 8s, 500 more iterations: 3h 30m 43s.
[2025-11-13 11:11:51,098][__main__][INFO] - Starting iteration 479.
[2025-11-13 11:11:51,101][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:11:51,101][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:12:01,620][__main__][INFO] - Number of regex retries in iteration 479: 0
[2025-11-13 11:12:01,620][__main__][INFO] - agents played in iteration 479 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:12:02,059][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,142][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:02,184][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:12:02,184][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:12:02,861][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:12:03,158][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:12:03,486][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:12:03,819][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:12:04,145][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:12:04,472][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:12:04,796][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:12:05,120][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:12:05,446][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:12:05,774][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:12:06,098][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:12:06,424][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:12:06,748][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:12:07,073][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:12:07,397][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:12:07,722][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:12:08,046][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:12:08,370][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:12:08,694][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:12:09,020][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:12:09,343][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:12:09,669][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:12:09,994][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:12:10,318][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:12:10,643][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:12:10,969][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:12:11,294][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:12:11,619][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:12:11,944][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:12:12,272][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:12:12,597][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:12:12,922][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:12:13,249][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:12:13,923][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:12:14,672][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:12:14,674][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:12:14,676][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:12:15,597][__main__][INFO] - Iteration 480 took 24s (42.94% Gen, 53.29% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 19m 27s. Estimated total time: 20h 24m 53s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 49s, 500 more iterations: 3h 24m 8s.
[2025-11-13 11:12:15,600][__main__][INFO] - Starting iteration 480.
[2025-11-13 11:12:15,603][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 47 and human policies 1.
[2025-11-13 11:12:15,603][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:12:26,557][__main__][INFO] - Number of regex retries in iteration 480: 0
[2025-11-13 11:12:26,558][__main__][INFO] - agents played in iteration 480 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:12:26,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:27,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:27,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:27,094][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:27,094][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:12:27,094][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:12:27,752][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:12:28,047][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:12:28,374][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:12:28,706][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:12:29,033][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:12:29,362][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:12:29,694][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:12:30,022][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:12:30,349][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:12:30,672][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:12:30,999][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:12:31,326][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:12:31,652][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:12:31,979][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:12:32,307][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:12:32,633][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:12:32,960][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:12:33,284][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:12:33,612][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:12:33,944][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:12:34,269][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:12:34,597][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:12:34,923][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:12:35,248][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:12:35,573][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:12:35,900][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:12:36,226][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:12:36,552][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:12:36,880][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:12:37,207][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:12:37,536][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:12:37,862][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:12:38,188][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:12:38,867][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:12:39,589][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:12:39,590][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:12:39,593][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:12:41,167][__main__][INFO] - Iteration 481 took 25s (42.85% Gen, 50.99% Train). Generation: 10s, Training: 13s. Estimated remaining time: 21h 12m 23s. Estimated total time: 21h 18m 15s. Time estimates for 10 more iterations: 4m 15s, 100 more iterations: 42m 36s, 500 more iterations: 3h 33m 2s.
[2025-11-13 11:12:41,170][__main__][INFO] - Starting iteration 481.
[2025-11-13 11:12:41,173][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:12:41,173][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:12:51,456][__main__][INFO] - Number of regex retries in iteration 481: 0
[2025-11-13 11:12:51,457][__main__][INFO] - agents played in iteration 481 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:12:51,869][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:52,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:52,264][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:52,305][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:12:52,305][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:12:52,306][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:12:52,986][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:12:53,283][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:12:53,612][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:12:53,935][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:12:54,260][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:12:54,587][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:12:54,912][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:12:55,236][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:12:55,561][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:12:55,885][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:12:56,210][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:12:56,533][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:12:56,857][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:12:57,184][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:12:57,508][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:12:57,832][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:12:58,156][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:12:58,482][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:12:58,805][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:12:59,133][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:12:59,458][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:12:59,783][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:13:00,108][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:13:00,431][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:13:00,755][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:13:01,080][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:13:01,407][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:13:01,735][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:13:02,062][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:13:02,388][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:13:02,716][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:13:03,045][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:13:03,372][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:13:04,052][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:13:04,752][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:13:04,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:13:04,756][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:13:05,602][__main__][INFO] - Iteration 482 took 24s (42.09% Gen, 54.44% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 15m 13s. Estimated total time: 20h 21m 30s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 43s, 500 more iterations: 3h 23m 35s.
[2025-11-13 11:13:05,604][__main__][INFO] - Starting iteration 482.
[2025-11-13 11:13:05,607][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:13:05,607][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:13:16,792][__main__][INFO] - Number of regex retries in iteration 482: 0
[2025-11-13 11:13:16,793][__main__][INFO] - agents played in iteration 482 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:13:17,216][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:17,257][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:17,297][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:17,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:17,338][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:13:17,338][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:13:18,030][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:13:18,328][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:13:18,653][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:13:18,980][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:13:19,305][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:13:19,636][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:13:19,966][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:13:20,289][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:13:20,617][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:13:20,942][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:13:21,270][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:13:21,594][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:13:21,922][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:13:22,248][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:13:22,573][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:13:22,898][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:13:23,226][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:13:23,551][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:13:23,875][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:13:24,205][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:13:24,533][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:13:24,858][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:13:25,185][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:13:25,512][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:13:25,839][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:13:26,165][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:13:26,494][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:13:26,821][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:13:27,148][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:13:27,473][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:13:27,800][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:13:28,125][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:13:28,452][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:13:29,200][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:13:29,902][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:13:29,904][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:13:29,905][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:13:30,729][__main__][INFO] - Iteration 483 took 25s (44.52% Gen, 52.19% Train). Generation: 11s, Training: 13s. Estimated remaining time: 20h 49m 29s. Estimated total time: 20h 56m 11s. Time estimates for 10 more iterations: 4m 11s, 100 more iterations: 41m 52s, 500 more iterations: 3h 29m 21s.
[2025-11-13 11:13:30,731][__main__][INFO] - Starting iteration 483.
[2025-11-13 11:13:30,735][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:13:30,735][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:13:40,831][__main__][INFO] - Number of regex retries in iteration 483: 0
[2025-11-13 11:13:40,832][__main__][INFO] - agents played in iteration 483 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:13:41,257][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:41,298][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:41,339][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:41,380][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:13:41,381][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:13:41,381][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:13:42,061][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:13:42,357][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:13:42,683][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:13:43,008][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:13:43,337][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:13:43,665][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:13:43,991][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:13:44,317][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:13:44,646][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:13:44,973][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:13:45,305][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:13:45,630][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:13:45,955][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:13:46,286][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:13:46,614][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:13:46,943][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:13:47,268][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:13:47,592][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:13:47,916][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:13:48,245][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:13:48,571][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:13:48,897][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:13:49,224][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:13:49,550][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:13:49,880][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:13:50,213][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:13:50,547][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:13:50,875][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:13:51,201][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:13:51,525][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:13:51,852][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:13:52,179][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:13:52,505][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:13:53,199][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:13:53,907][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:13:53,908][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:13:53,910][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:13:54,767][__main__][INFO] - Iteration 484 took 24s (42.01% Gen, 54.42% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 54m 35s. Estimated total time: 20h 1m 41s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 3s, 500 more iterations: 3h 20m 16s.
[2025-11-13 11:13:54,769][__main__][INFO] - Starting iteration 484.
[2025-11-13 11:13:54,792][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:13:54,792][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:14:05,359][__main__][INFO] - Number of regex retries in iteration 484: 0
[2025-11-13 11:14:05,360][__main__][INFO] - agents played in iteration 484 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:14:05,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:05,813][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:05,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:05,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:05,895][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:14:05,896][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:14:06,592][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:14:06,890][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:14:07,218][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:14:07,545][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:14:07,873][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:14:08,198][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:14:08,523][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:14:08,848][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:14:09,174][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:14:09,503][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:14:09,828][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:14:10,153][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:14:10,480][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:14:10,804][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:14:11,129][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:14:11,463][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:14:11,789][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:14:12,117][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:14:12,447][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:14:12,780][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:14:13,104][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:14:13,432][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:14:13,758][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:14:14,086][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:14:14,415][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:14:14,744][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:14:15,074][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:14:15,405][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:14:15,733][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:14:16,062][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:14:16,392][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:14:16,718][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:14:17,046][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:14:17,750][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:14:18,451][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:14:18,453][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:14:18,454][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:14:19,265][__main__][INFO] - Iteration 485 took 24s (43.14% Gen, 53.46% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 17m 13s. Estimated total time: 20h 24m 43s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 49s, 500 more iterations: 3h 24m 7s.
[2025-11-13 11:14:19,268][__main__][INFO] - Starting iteration 485.
[2025-11-13 11:14:19,270][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:14:19,271][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:14:28,975][__main__][INFO] - Number of regex retries in iteration 485: 0
[2025-11-13 11:14:28,976][__main__][INFO] - agents played in iteration 485 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:14:29,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:29,451][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:29,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:29,531][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:29,532][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:14:29,533][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:14:30,234][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:14:30,531][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:14:30,858][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:14:31,184][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:14:31,509][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:14:31,834][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:14:32,158][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:14:32,484][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:14:32,809][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:14:33,134][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:14:33,458][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:14:33,783][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:14:34,108][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:14:34,433][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:14:34,761][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:14:35,086][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:14:35,413][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:14:35,738][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:14:36,064][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:14:36,389][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:14:36,714][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:14:37,040][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:14:37,369][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:14:37,696][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:14:38,022][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:14:38,351][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:14:38,680][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:14:39,008][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:14:39,335][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:14:39,661][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:14:39,987][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:14:40,312][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:14:40,638][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:14:41,335][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:14:42,044][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:14:42,046][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:14:42,047][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:14:42,859][__main__][INFO] - Iteration 486 took 23s (41.13% Gen, 55.41% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 31m 35s. Estimated total time: 19h 39m 28s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 18s, 500 more iterations: 3h 16m 34s.
[2025-11-13 11:14:42,861][__main__][INFO] - Starting iteration 486.
[2025-11-13 11:14:42,864][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:14:42,865][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:14:52,859][__main__][INFO] - Number of regex retries in iteration 486: 0
[2025-11-13 11:14:52,860][__main__][INFO] - agents played in iteration 486 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:14:53,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:53,328][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:53,368][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:53,408][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:14:53,408][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:14:53,409][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:14:54,112][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:14:54,409][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:14:54,736][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:14:55,062][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:14:55,391][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:14:55,715][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:14:56,040][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:14:56,365][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:14:56,691][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:14:57,017][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:14:57,343][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:14:57,669][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:14:57,994][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:14:58,320][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:14:58,648][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:14:58,974][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:14:59,301][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:14:59,627][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:14:59,952][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:15:00,277][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:15:00,603][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:15:00,929][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:15:01,256][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:15:01,583][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:15:01,908][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:15:02,235][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:15:02,565][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:15:02,894][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:15:03,222][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:15:03,549][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:15:03,877][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:15:04,203][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:15:04,530][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:15:05,243][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:15:05,949][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:15:05,950][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:15:05,952][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:15:06,778][__main__][INFO] - Iteration 487 took 23s (41.79% Gen, 54.75% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 47m 26s. Estimated total time: 19h 55m 44s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 51s, 500 more iterations: 3h 19m 17s.
[2025-11-13 11:15:06,780][__main__][INFO] - Starting iteration 487.
[2025-11-13 11:15:06,783][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:15:06,784][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:15:16,983][__main__][INFO] - Number of regex retries in iteration 487: 0
[2025-11-13 11:15:16,984][__main__][INFO] - agents played in iteration 487 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:15:17,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:17,468][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:17,509][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:17,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:17,549][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:15:17,549][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:15:18,244][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:15:18,540][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:15:18,865][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:15:19,193][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:15:19,522][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:15:19,849][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:15:20,176][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:15:20,501][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:15:20,827][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:15:21,158][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:15:21,484][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:15:21,809][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:15:22,135][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:15:22,465][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:15:22,793][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:15:23,128][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:15:23,456][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:15:23,790][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:15:24,119][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:15:24,448][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:15:24,775][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:15:25,106][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:15:25,435][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:15:25,769][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:15:26,098][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:15:26,431][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:15:26,756][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:15:27,085][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:15:27,412][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:15:27,738][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:15:28,065][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:15:28,392][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:15:28,718][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:15:29,405][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:15:30,120][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:15:30,121][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:15:30,122][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:15:30,984][__main__][INFO] - Iteration 488 took 24s (42.14% Gen, 54.29% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 1m 23s. Estimated total time: 20h 10m 5s. Time estimates for 10 more iterations: 4m 2s, 100 more iterations: 40m 20s, 500 more iterations: 3h 21m 40s.
[2025-11-13 11:15:30,986][__main__][INFO] - Starting iteration 488.
[2025-11-13 11:15:30,990][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:15:30,990][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:15:40,844][__main__][INFO] - Number of regex retries in iteration 488: 0
[2025-11-13 11:15:40,845][__main__][INFO] - agents played in iteration 488 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:15:41,267][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:41,308][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:41,349][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:41,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:15:41,390][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:15:41,390][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:15:42,077][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:15:42,372][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:15:42,697][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:15:43,022][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:15:43,347][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:15:43,671][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:15:43,997][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:15:44,321][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:15:44,645][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:15:44,973][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:15:45,297][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:15:45,621][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:15:45,946][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:15:46,270][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:15:46,595][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:15:46,920][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:15:47,246][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:15:47,572][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:15:47,898][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:15:48,223][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:15:48,549][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:15:48,876][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:15:49,202][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:15:49,527][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:15:49,855][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:15:50,181][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:15:50,511][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:15:50,838][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:15:51,165][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:15:51,490][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:15:51,816][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:15:52,142][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:15:52,470][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:15:53,167][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:15:53,867][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:15:53,869][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:15:53,870][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:15:54,682][__main__][INFO] - Iteration 489 took 23s (41.59% Gen, 54.97% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 35m 34s. Estimated total time: 19h 44m 40s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 29s, 500 more iterations: 3h 17m 26s.
[2025-11-13 11:15:54,684][__main__][INFO] - Starting iteration 489.
[2025-11-13 11:15:54,688][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:15:54,688][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:16:03,963][__main__][INFO] - Number of regex retries in iteration 489: 0
[2025-11-13 11:16:03,963][__main__][INFO] - agents played in iteration 489 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:16:04,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:04,416][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:04,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:04,496][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:04,496][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:16:04,497][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:16:05,194][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:16:05,491][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:16:05,819][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:16:06,154][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:16:06,487][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:16:06,813][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:16:07,140][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:16:07,466][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:16:07,792][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:16:08,116][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:16:08,441][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:16:08,766][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:16:09,091][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:16:09,415][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:16:09,740][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:16:10,064][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:16:10,388][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:16:10,713][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:16:11,039][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:16:11,365][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:16:11,693][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:16:12,020][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:16:12,346][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:16:12,676][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:16:13,004][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:16:13,330][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:16:13,658][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:16:13,984][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:16:14,316][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:16:14,642][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:16:14,969][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:16:15,294][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:16:15,622][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:16:16,312][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:16:17,007][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:16:17,008][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:16:17,010][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:16:17,817][__main__][INFO] - Iteration 490 took 23s (40.10% Gen, 56.41% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 7m 1s. Estimated total time: 19h 16m 30s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 45s.
[2025-11-13 11:16:17,819][__main__][INFO] - Starting iteration 490.
[2025-11-13 11:16:17,821][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 48 and human policies 1.
[2025-11-13 11:16:17,822][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:16:27,377][__main__][INFO] - Number of regex retries in iteration 490: 0
[2025-11-13 11:16:27,378][__main__][INFO] - agents played in iteration 490 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:16:27,791][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:27,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:27,871][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:27,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:27,911][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:16:27,912][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:16:28,595][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:16:29,026][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:16:29,338][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:16:29,665][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:16:29,995][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:16:30,323][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:16:30,647][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:16:30,975][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:16:31,299][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:16:31,626][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:16:31,950][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:16:32,278][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:16:32,610][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:16:32,935][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:16:33,263][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:16:33,587][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:16:33,912][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:16:34,236][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:16:34,561][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:16:34,889][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:16:35,216][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:16:35,548][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:16:35,876][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:16:36,203][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:16:36,529][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:16:36,855][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:16:37,181][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:16:37,508][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:16:37,834][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:16:38,161][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:16:38,488][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:16:38,814][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:16:39,142][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:16:39,869][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:16:40,622][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:16:40,623][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:16:40,625][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:16:42,249][__main__][INFO] - Iteration 491 took 24s (39.12% Gen, 54.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 20h 11m 33s. Estimated total time: 20h 21m 26s. Time estimates for 10 more iterations: 4m 4s, 100 more iterations: 40m 42s, 500 more iterations: 3h 23m 34s.
[2025-11-13 11:16:42,251][__main__][INFO] - Starting iteration 491.
[2025-11-13 11:16:42,254][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:16:42,255][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:16:51,159][__main__][INFO] - Number of regex retries in iteration 491: 0
[2025-11-13 11:16:51,160][__main__][INFO] - agents played in iteration 491 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:16:51,579][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:51,619][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:51,658][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:51,698][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:16:51,698][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:16:51,699][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:16:52,391][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:16:52,688][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:16:53,016][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:16:53,343][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:16:53,668][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:16:53,992][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:16:54,318][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:16:54,642][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:16:54,968][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:16:55,292][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:16:55,617][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:16:55,945][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:16:56,270][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:16:56,595][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:16:56,921][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:16:57,245][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:16:57,572][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:16:57,906][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:16:58,238][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:16:58,567][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:16:58,897][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:16:59,226][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:16:59,552][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:16:59,880][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:17:00,209][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:17:00,538][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:17:00,865][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:17:01,198][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:17:01,532][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:17:01,858][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:17:02,186][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:17:02,513][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:17:02,838][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:17:03,526][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:17:04,240][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:17:04,241][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:17:04,243][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:17:05,155][__main__][INFO] - Iteration 492 took 22s (38.88% Gen, 57.13% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 54m 49s. Estimated total time: 19h 5m 5s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 10s, 500 more iterations: 3h 10m 50s.
[2025-11-13 11:17:05,157][__main__][INFO] - Starting iteration 492.
[2025-11-13 11:17:05,161][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:17:05,161][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:17:14,919][__main__][INFO] - Number of regex retries in iteration 492: 0
[2025-11-13 11:17:14,920][__main__][INFO] - agents played in iteration 492 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:17:15,350][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:15,390][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:15,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:15,469][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:15,469][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:17:15,469][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:17:16,137][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:17:16,434][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:17:16,760][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:17:17,086][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:17:17,415][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:17:17,744][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:17:18,073][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:17:18,398][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:17:18,726][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:17:19,059][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:17:19,387][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:17:19,714][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:17:20,041][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:17:20,374][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:17:20,704][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:17:21,036][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:17:21,363][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:17:21,688][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:17:22,018][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:17:22,346][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:17:22,673][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:17:23,003][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:17:23,331][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:17:23,658][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:17:23,986][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:17:24,311][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:17:24,639][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:17:24,971][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:17:25,299][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:17:25,629][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:17:25,956][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:17:26,282][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:17:26,609][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:17:27,314][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:17:28,023][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:17:28,025][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:17:28,026][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:17:28,871][__main__][INFO] - Iteration 493 took 23s (41.15% Gen, 55.27% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 34m 55s. Estimated total time: 19h 45m 35s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 31s, 500 more iterations: 3h 17m 35s.
[2025-11-13 11:17:28,873][__main__][INFO] - Starting iteration 493.
[2025-11-13 11:17:28,876][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:17:28,877][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:17:37,820][__main__][INFO] - Number of regex retries in iteration 493: 0
[2025-11-13 11:17:37,821][__main__][INFO] - agents played in iteration 493 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:17:38,238][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:38,280][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:38,319][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:38,359][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:17:38,360][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:17:38,360][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:17:39,035][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:17:39,332][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:17:39,663][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:17:39,988][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:17:40,314][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:17:40,639][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:17:40,965][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:17:41,290][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:17:41,616][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:17:41,946][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:17:42,274][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:17:42,602][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:17:42,926][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:17:43,250][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:17:43,579][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:17:43,909][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:17:44,233][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:17:44,561][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:17:44,886][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:17:45,211][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:17:45,536][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:17:45,864][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:17:46,193][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:17:46,522][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:17:46,849][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:17:47,174][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:17:47,501][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:17:47,828][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:17:48,157][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:17:48,486][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:17:48,813][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:17:49,140][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:17:49,467][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:17:50,170][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:17:50,861][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:17:50,863][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:17:50,865][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:17:51,831][__main__][INFO] - Iteration 494 took 22s (38.96% Gen, 56.82% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 56m 43s. Estimated total time: 19h 7m 46s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 17s.
[2025-11-13 11:17:51,833][__main__][INFO] - Starting iteration 494.
[2025-11-13 11:17:51,836][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:17:51,836][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:17:58,627][mllm.models.large_language_model_local][WARNING] - Response <a> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 11:18:02,374][__main__][INFO] - Number of regex retries in iteration 494: 1
[2025-11-13 11:18:02,374][__main__][INFO] - agents played in iteration 494 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:18:02,797][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:02,833][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:02,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:02,898][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:02,899][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:18:02,900][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:18:03,559][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:18:03,855][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:18:04,179][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:18:04,507][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:18:04,830][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:18:05,156][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:18:05,482][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:18:05,808][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:18:06,135][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:18:06,458][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:18:06,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:18:07,104][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:18:07,427][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:18:07,752][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:18:08,078][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:18:08,401][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:18:08,724][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:18:09,047][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:18:09,371][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:18:09,695][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:18:10,020][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:18:10,343][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:18:10,670][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:18:10,995][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:18:11,320][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:18:11,645][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:18:11,972][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:18:12,295][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:18:12,619][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:18:12,944][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:18:13,267][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:18:13,589][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:18:13,912][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:18:14,608][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:18:15,295][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:18:15,296][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:18:15,297][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:18:16,160][__main__][INFO] - Iteration 495 took 24s (43.32% Gen, 53.13% Train). Generation: 10s, Training: 12s. Estimated remaining time: 20h 4m 47s. Estimated total time: 20h 16m 15s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 32s, 500 more iterations: 3h 22m 42s.
[2025-11-13 11:18:16,161][__main__][INFO] - Starting iteration 495.
[2025-11-13 11:18:16,164][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:18:16,164][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:18:25,572][__main__][INFO] - Number of regex retries in iteration 495: 0
[2025-11-13 11:18:25,573][__main__][INFO] - agents played in iteration 495 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:18:26,024][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:26,059][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:26,092][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:26,124][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:26,124][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:18:26,125][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:18:26,788][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:18:27,082][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:18:27,410][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:18:27,735][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:18:28,061][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:18:28,384][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:18:28,709][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:18:29,032][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:18:29,355][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:18:29,677][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:18:29,999][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:18:30,322][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:18:30,644][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:18:30,967][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:18:31,291][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:18:31,614][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:18:31,936][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:18:32,260][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:18:32,583][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:18:32,906][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:18:33,231][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:18:33,555][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:18:33,879][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:18:34,202][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:18:34,526][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:18:34,853][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:18:35,178][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:18:35,502][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:18:35,827][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:18:36,152][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:18:36,476][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:18:36,801][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:18:37,127][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:18:37,817][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:18:38,540][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:18:38,541][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:18:38,542][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:18:39,389][__main__][INFO] - Iteration 496 took 23s (40.51% Gen, 55.84% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 9m 27s. Estimated total time: 19h 21m 17s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 42s, 500 more iterations: 3h 13m 32s.
[2025-11-13 11:18:39,391][__main__][INFO] - Starting iteration 496.
[2025-11-13 11:18:39,394][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:18:39,394][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:18:49,125][__main__][INFO] - Number of regex retries in iteration 496: 0
[2025-11-13 11:18:49,126][__main__][INFO] - agents played in iteration 496 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:18:49,550][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:49,583][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:49,616][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:49,648][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:18:49,649][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:18:49,649][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:18:50,328][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:18:50,623][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:18:50,949][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:18:51,273][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:18:51,596][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:18:51,920][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:18:52,243][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:18:52,566][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:18:52,890][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:18:53,213][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:18:53,537][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:18:53,860][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:18:54,186][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:18:54,510][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:18:54,834][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:18:55,159][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:18:55,485][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:18:55,811][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:18:56,137][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:18:56,461][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:18:56,784][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:18:57,110][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:18:57,435][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:18:57,762][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:18:58,090][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:18:58,414][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:18:58,738][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:18:59,061][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:18:59,385][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:18:59,708][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:19:00,030][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:19:00,353][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:19:00,678][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:19:01,374][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:19:02,057][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:19:02,058][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:19:02,060][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:19:02,895][__main__][INFO] - Iteration 497 took 23s (41.40% Gen, 55.03% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 22m 52s. Estimated total time: 19h 35m 6s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 51s.
[2025-11-13 11:19:02,897][__main__][INFO] - Starting iteration 497.
[2025-11-13 11:19:02,900][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:19:02,900][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:19:12,268][__main__][INFO] - Number of regex retries in iteration 497: 0
[2025-11-13 11:19:12,269][__main__][INFO] - agents played in iteration 497 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:19:12,690][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:12,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:12,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:12,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:12,789][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:19:12,789][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:19:13,480][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:19:13,775][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:19:14,101][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:19:14,426][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:19:14,748][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:19:15,070][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:19:15,392][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:19:15,714][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:19:16,037][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:19:16,359][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:19:16,683][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:19:17,007][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:19:17,329][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:19:17,651][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:19:17,974][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:19:18,296][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:19:18,623][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:19:18,945][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:19:19,268][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:19:19,592][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:19:19,914][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:19:20,237][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:19:20,561][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:19:20,885][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:19:21,211][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:19:21,536][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:19:21,861][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:19:22,187][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:19:22,512][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:19:22,836][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:19:23,159][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:19:23,482][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:19:23,807][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:19:24,502][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:19:25,186][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:19:25,188][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:19:25,189][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:19:25,996][__main__][INFO] - Iteration 498 took 23s (40.56% Gen, 55.94% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 2m 14s. Estimated total time: 19h 14m 51s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 29s, 500 more iterations: 3h 12m 28s.
[2025-11-13 11:19:25,998][__main__][INFO] - Starting iteration 498.
[2025-11-13 11:19:26,001][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:19:26,002][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:19:35,880][__main__][INFO] - Number of regex retries in iteration 498: 0
[2025-11-13 11:19:35,880][__main__][INFO] - agents played in iteration 498 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:19:36,316][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:36,352][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:36,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:36,421][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:19:36,422][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:19:36,422][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:19:37,105][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:19:37,400][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:19:37,724][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:19:38,047][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:19:38,371][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:19:38,693][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:19:39,021][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:19:39,345][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:19:39,671][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:19:39,997][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:19:40,318][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:19:40,647][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:19:40,976][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:19:41,298][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:19:41,620][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:19:41,944][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:19:42,268][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:19:42,591][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:19:42,915][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:19:43,241][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:19:43,567][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:19:43,891][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:19:44,217][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:19:44,544][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:19:44,868][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:19:45,193][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:19:45,518][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:19:45,842][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:19:46,167][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:19:46,491][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:19:46,816][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:19:47,138][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:19:47,462][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:19:48,155][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:19:48,842][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:19:48,843][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:19:48,845][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:19:49,716][__main__][INFO] - Iteration 499 took 23s (41.65% Gen, 54.67% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 32m 47s. Estimated total time: 19h 45m 47s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 31s, 500 more iterations: 3h 17m 37s.
[2025-11-13 11:19:49,718][__main__][INFO] - Starting iteration 499.
[2025-11-13 11:19:49,721][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:19:49,721][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:19:59,792][__main__][INFO] - Number of regex retries in iteration 499: 0
[2025-11-13 11:19:59,792][__main__][INFO] - agents played in iteration 499 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:20:00,224][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:00,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:00,294][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:00,327][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:00,328][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:20:00,328][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:20:01,046][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:20:01,339][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:20:01,664][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:20:01,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:20:02,317][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:20:02,641][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:20:02,965][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:20:03,293][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:20:03,619][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:20:03,943][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:20:04,265][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:20:04,589][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:20:04,911][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:20:05,231][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:20:05,553][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:20:05,876][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:20:06,199][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:20:06,522][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:20:06,845][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:20:07,169][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:20:07,495][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:20:07,817][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:20:08,141][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:20:08,466][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:20:08,789][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:20:09,116][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:20:09,439][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:20:09,763][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:20:10,086][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:20:10,411][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:20:10,734][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:20:11,056][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:20:11,380][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:20:12,084][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:20:12,770][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:20:12,772][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:20:12,773][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:20:13,563][__main__][INFO] - Iteration 500 took 23s (42.24% Gen, 54.44% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 38m 45s. Estimated total time: 19h 52m 10s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 44s, 500 more iterations: 3h 18m 41s.
[2025-11-13 11:20:13,565][__main__][INFO] - Starting iteration 500.
[2025-11-13 11:20:13,568][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 49 and human policies 1.
[2025-11-13 11:20:13,569][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:20:22,779][__main__][INFO] - Number of regex retries in iteration 500: 0
[2025-11-13 11:20:22,779][__main__][INFO] - agents played in iteration 500 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:20:23,201][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:23,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:23,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:23,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:23,304][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:20:23,304][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:20:24,014][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:20:24,308][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:20:24,634][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:20:24,961][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:20:25,284][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:20:25,610][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:20:25,932][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:20:26,256][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:20:26,577][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:20:26,899][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:20:27,223][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:20:27,547][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:20:27,870][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:20:28,195][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:20:28,520][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:20:28,843][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:20:29,166][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:20:29,489][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:20:29,810][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:20:30,135][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:20:30,459][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:20:30,781][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:20:31,107][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:20:31,432][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:20:31,757][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:20:32,082][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:20:32,412][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:20:32,738][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:20:33,061][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:20:33,384][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:20:33,709][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:20:34,032][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:20:34,357][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:20:35,060][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:20:35,754][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:20:35,755][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:20:35,757][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:20:37,318][__main__][INFO] - Iteration 501 took 23s (38.78% Gen, 54.64% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 33m 45s. Estimated total time: 19h 47m 33s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 35s, 500 more iterations: 3h 17m 55s.
[2025-11-13 11:20:37,320][__main__][INFO] - Starting iteration 501.
[2025-11-13 11:20:37,323][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:20:37,323][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:20:46,887][__main__][INFO] - Number of regex retries in iteration 501: 0
[2025-11-13 11:20:46,887][__main__][INFO] - agents played in iteration 501 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:20:47,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:47,347][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:47,381][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:47,415][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:20:47,416][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:20:47,416][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:20:48,124][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:20:48,417][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:20:48,741][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:20:49,064][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:20:49,387][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:20:49,711][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:20:50,035][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:20:50,357][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:20:50,681][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:20:51,006][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:20:51,334][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:20:51,659][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:20:51,986][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:20:52,309][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:20:52,635][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:20:52,961][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:20:53,286][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:20:53,609][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:20:53,933][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:20:54,254][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:20:54,578][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:20:54,902][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:20:55,226][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:20:55,550][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:20:55,876][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:20:56,199][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:20:56,522][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:20:56,844][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:20:57,167][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:20:57,489][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:20:57,812][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:20:58,136][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:20:58,461][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:20:59,168][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:20:59,867][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:20:59,868][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:20:59,870][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:21:00,687][__main__][INFO] - Iteration 502 took 23s (40.93% Gen, 55.56% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 14m 3s. Estimated total time: 19h 28m 15s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 56s, 500 more iterations: 3h 14m 42s.
[2025-11-13 11:21:00,689][__main__][INFO] - Starting iteration 502.
[2025-11-13 11:21:00,692][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:21:00,693][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:21:10,282][__main__][INFO] - Number of regex retries in iteration 502: 0
[2025-11-13 11:21:10,283][__main__][INFO] - agents played in iteration 502 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:21:10,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:10,750][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:10,783][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:10,817][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:10,817][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:21:10,817][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:21:11,508][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:21:11,803][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:21:12,131][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:21:12,452][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:21:12,773][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:21:13,095][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:21:13,418][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:21:13,739][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:21:14,063][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:21:14,386][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:21:14,709][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:21:15,031][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:21:15,354][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:21:15,679][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:21:16,003][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:21:16,326][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:21:16,648][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:21:16,971][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:21:17,295][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:21:17,619][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:21:17,949][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:21:18,274][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:21:18,600][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:21:18,925][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:21:19,249][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:21:19,574][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:21:19,897][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:21:20,220][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:21:20,544][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:21:20,867][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:21:21,191][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:21:21,516][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:21:21,839][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:21:22,532][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:21:23,213][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:21:23,214][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:21:23,216][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:21:24,278][__main__][INFO] - Iteration 503 took 23s (40.66% Gen, 54.83% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 24m 44s. Estimated total time: 19h 39m 20s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 18s, 500 more iterations: 3h 16m 33s.
[2025-11-13 11:21:24,280][__main__][INFO] - Starting iteration 503.
[2025-11-13 11:21:24,283][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:21:24,283][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:21:33,649][__main__][INFO] - Number of regex retries in iteration 503: 0
[2025-11-13 11:21:33,649][__main__][INFO] - agents played in iteration 503 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:21:34,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:34,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:34,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:34,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:34,181][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:21:34,181][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:21:34,879][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:21:35,172][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:21:35,497][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:21:35,826][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:21:36,149][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:21:36,471][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:21:36,800][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:21:37,133][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:21:37,454][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:21:37,777][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:21:38,098][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:21:38,421][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:21:38,744][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:21:39,066][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:21:39,390][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:21:39,712][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:21:40,037][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:21:40,361][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:21:40,686][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:21:41,011][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:21:41,339][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:21:41,661][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:21:41,986][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:21:42,311][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:21:42,635][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:21:42,958][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:21:43,287][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:21:43,610][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:21:43,933][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:21:44,257][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:21:44,581][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:21:44,906][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:21:45,230][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:21:45,929][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:21:46,608][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:21:46,609][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:21:46,610][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:21:47,411][__main__][INFO] - Iteration 504 took 23s (40.49% Gen, 56.04% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 1m 28s. Estimated total time: 19h 16m 27s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 32s, 500 more iterations: 3h 12m 44s.
[2025-11-13 11:21:47,413][__main__][INFO] - Starting iteration 504.
[2025-11-13 11:21:47,416][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:21:47,416][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:21:56,757][__main__][INFO] - Number of regex retries in iteration 504: 0
[2025-11-13 11:21:56,758][__main__][INFO] - agents played in iteration 504 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:21:57,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:57,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:57,246][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:57,279][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:21:57,280][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:21:57,280][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:21:57,952][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:21:58,248][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:21:58,573][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:21:58,894][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:21:59,216][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:21:59,540][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:21:59,868][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:22:00,193][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:22:00,516][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:22:00,839][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:22:01,161][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:22:01,483][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:22:01,804][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:22:02,127][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:22:02,453][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:22:02,776][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:22:03,105][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:22:03,431][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:22:03,760][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:22:04,086][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:22:04,412][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:22:04,738][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:22:05,062][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:22:05,388][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:22:05,712][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:22:06,035][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:22:06,359][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:22:06,683][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:22:07,008][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:22:07,332][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:22:07,656][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:22:07,978][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:22:08,301][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:22:09,006][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:22:09,688][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:22:09,689][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:22:09,691][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:22:10,502][__main__][INFO] - Iteration 505 took 23s (40.46% Gen, 56.02% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 59m 0s. Estimated total time: 19h 14m 22s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 23s.
[2025-11-13 11:22:10,505][__main__][INFO] - Starting iteration 505.
[2025-11-13 11:22:10,507][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:22:10,508][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:22:19,787][__main__][INFO] - Number of regex retries in iteration 505: 0
[2025-11-13 11:22:19,788][__main__][INFO] - agents played in iteration 505 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:22:20,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:20,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:20,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:20,309][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:20,310][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:22:20,310][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:22:20,987][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:22:21,283][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:22:21,608][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:22:21,929][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:22:22,252][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:22:22,575][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:22:22,898][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:22:23,221][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:22:23,545][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:22:23,869][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:22:24,193][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:22:24,516][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:22:24,839][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:22:25,161][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:22:25,483][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:22:25,808][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:22:26,131][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:22:26,453][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:22:26,777][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:22:27,100][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:22:27,423][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:22:27,746][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:22:28,073][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:22:28,399][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:22:28,722][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:22:29,047][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:22:29,372][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:22:29,697][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:22:30,021][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:22:30,345][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:22:30,671][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:22:30,995][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:22:31,317][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:22:32,013][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:22:32,688][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:22:32,690][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:22:32,691][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:22:33,521][__main__][INFO] - Iteration 506 took 23s (40.32% Gen, 56.07% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 54m 59s. Estimated total time: 19h 10m 43s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 21s, 500 more iterations: 3h 11m 47s.
[2025-11-13 11:22:33,523][__main__][INFO] - Starting iteration 506.
[2025-11-13 11:22:33,526][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:22:33,527][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:22:42,408][__main__][INFO] - Number of regex retries in iteration 506: 0
[2025-11-13 11:22:42,408][__main__][INFO] - agents played in iteration 506 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:22:42,832][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:42,866][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:42,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:42,932][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:22:42,933][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:22:42,934][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:22:43,613][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:22:43,906][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:22:44,229][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:22:44,553][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:22:44,876][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:22:45,199][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:22:45,524][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:22:45,848][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:22:46,175][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:22:46,504][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:22:46,827][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:22:47,149][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:22:47,478][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:22:47,801][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:22:48,126][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:22:48,451][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:22:48,777][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:22:49,101][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:22:49,432][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:22:49,755][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:22:50,077][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:22:50,401][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:22:50,726][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:22:51,048][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:22:51,372][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:22:51,696][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:22:52,021][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:22:52,342][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:22:52,667][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:22:52,990][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:22:53,315][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:22:53,639][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:22:53,963][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:22:54,663][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:22:55,344][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:22:55,346][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:22:55,347][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:22:56,152][__main__][INFO] - Iteration 507 took 22s (39.25% Gen, 57.18% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 35m 14s. Estimated total time: 18h 51m 21s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 42s, 500 more iterations: 3h 8m 33s.
[2025-11-13 11:22:56,155][__main__][INFO] - Starting iteration 507.
[2025-11-13 11:22:56,158][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:22:56,159][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:23:05,275][__main__][INFO] - Number of regex retries in iteration 507: 0
[2025-11-13 11:23:05,275][__main__][INFO] - agents played in iteration 507 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:23:05,709][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:05,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:05,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:05,809][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:05,809][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:23:05,810][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:23:06,485][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:23:06,778][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:23:07,101][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:23:07,424][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:23:07,749][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:23:08,069][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:23:08,390][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:23:08,713][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:23:09,035][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:23:09,360][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:23:09,684][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:23:10,006][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:23:10,327][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:23:10,649][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:23:10,973][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:23:11,296][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:23:11,623][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:23:11,946][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:23:12,268][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:23:12,591][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:23:12,914][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:23:13,238][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:23:13,563][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:23:13,888][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:23:14,211][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:23:14,534][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:23:14,859][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:23:15,183][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:23:15,506][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:23:15,831][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:23:16,154][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:23:16,479][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:23:16,803][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:23:17,510][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:23:18,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:23:18,203][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:23:18,204][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:23:19,109][__main__][INFO] - Iteration 508 took 22s (39.72% Gen, 56.33% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 51m 5s. Estimated total time: 19h 7m 35s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 15s, 500 more iterations: 3h 11m 15s.
[2025-11-13 11:23:19,111][__main__][INFO] - Starting iteration 508.
[2025-11-13 11:23:19,114][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:23:19,115][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:23:28,362][__main__][INFO] - Number of regex retries in iteration 508: 0
[2025-11-13 11:23:28,362][__main__][INFO] - agents played in iteration 508 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:23:28,788][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:28,823][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:28,856][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:28,891][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:28,891][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:23:28,892][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:23:29,563][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:23:29,856][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:23:30,179][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:23:30,501][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:23:30,824][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:23:31,149][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:23:31,471][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:23:31,793][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:23:32,120][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:23:32,443][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:23:32,768][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:23:33,090][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:23:33,412][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:23:33,733][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:23:34,061][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:23:34,382][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:23:34,707][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:23:35,030][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:23:35,352][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:23:35,675][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:23:36,000][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:23:36,325][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:23:36,651][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:23:36,975][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:23:37,299][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:23:37,624][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:23:37,947][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:23:38,271][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:23:38,595][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:23:38,919][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:23:39,243][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:23:39,567][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:23:39,890][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:23:40,595][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:23:41,277][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:23:41,278][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:23:41,280][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:23:42,095][__main__][INFO] - Iteration 509 took 22s (40.24% Gen, 56.21% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 52m 11s. Estimated total time: 19h 9m 4s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 18s, 500 more iterations: 3h 11m 30s.
[2025-11-13 11:23:42,096][__main__][INFO] - Starting iteration 509.
[2025-11-13 11:23:42,099][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:23:42,100][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:23:50,916][__main__][INFO] - Number of regex retries in iteration 509: 0
[2025-11-13 11:23:50,916][__main__][INFO] - agents played in iteration 509 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:23:51,343][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:51,376][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:51,409][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:51,442][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:23:51,443][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:23:51,443][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:23:52,120][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:23:52,413][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:23:52,737][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:23:53,061][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:23:53,383][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:23:53,707][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:23:54,030][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:23:54,356][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:23:54,685][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:23:55,009][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:23:55,336][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:23:55,659][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:23:55,983][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:23:56,309][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:23:56,634][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:23:56,960][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:23:57,286][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:23:57,615][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:23:57,945][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:23:58,269][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:23:58,593][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:23:58,917][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:23:59,241][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:23:59,566][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:23:59,890][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:24:00,213][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:24:00,537][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:24:00,861][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:24:01,185][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:24:01,510][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:24:01,834][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:24:02,158][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:24:02,481][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:24:03,178][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:24:03,862][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:24:03,863][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:24:03,865][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:24:04,682][__main__][INFO] - Iteration 510 took 22s (39.04% Gen, 57.34% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 31m 54s. Estimated total time: 18h 49m 10s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 38s, 500 more iterations: 3h 8m 11s.
[2025-11-13 11:24:04,684][__main__][INFO] - Starting iteration 510.
[2025-11-13 11:24:04,686][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 50 and human policies 1.
[2025-11-13 11:24:04,687][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:24:14,348][__main__][INFO] - Number of regex retries in iteration 510: 0
[2025-11-13 11:24:14,349][__main__][INFO] - agents played in iteration 510 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:24:14,774][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:14,809][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:14,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:14,875][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:14,876][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:24:14,876][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:24:15,556][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:24:15,850][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:24:16,174][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:24:16,500][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:24:16,823][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:24:17,151][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:24:17,480][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:24:17,805][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:24:18,130][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:24:18,457][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:24:18,781][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:24:19,105][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:24:19,428][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:24:19,751][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:24:20,080][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:24:20,407][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:24:20,733][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:24:21,057][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:24:21,379][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:24:21,706][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:24:22,033][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:24:22,356][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:24:22,680][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:24:23,004][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:24:23,327][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:24:23,657][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:24:23,981][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:24:24,306][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:24:24,629][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:24:24,951][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:24:25,274][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:24:25,598][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:24:25,922][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:24:26,638][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:24:27,335][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:24:27,337][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:24:27,339][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:24:28,987][__main__][INFO] - Iteration 511 took 24s (39.76% Gen, 53.45% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 57m 25s. Estimated total time: 20h 15m 5s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 30s, 500 more iterations: 3h 22m 30s.
[2025-11-13 11:24:28,989][__main__][INFO] - Starting iteration 511.
[2025-11-13 11:24:28,993][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:24:28,994][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:24:38,266][__main__][INFO] - Number of regex retries in iteration 511: 0
[2025-11-13 11:24:38,267][__main__][INFO] - agents played in iteration 511 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:24:38,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:38,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:38,756][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:38,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:24:38,790][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:24:38,790][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:24:39,475][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:24:39,769][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:24:40,094][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:24:40,416][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:24:40,740][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:24:41,061][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:24:41,385][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:24:41,708][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:24:42,031][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:24:42,354][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:24:42,678][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:24:43,000][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:24:43,326][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:24:43,650][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:24:43,974][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:24:44,296][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:24:44,624][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:24:44,950][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:24:45,278][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:24:45,605][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:24:45,929][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:24:46,253][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:24:46,577][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:24:46,900][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:24:47,224][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:24:47,549][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:24:47,874][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:24:48,198][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:24:48,522][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:24:48,845][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:24:49,170][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:24:49,494][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:24:49,818][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:24:50,516][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:24:51,190][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:24:51,191][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:24:51,193][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:24:52,073][__main__][INFO] - Iteration 512 took 23s (40.18% Gen, 56.00% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 55m 57s. Estimated total time: 19h 14m 0s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 20s.
[2025-11-13 11:24:52,075][__main__][INFO] - Starting iteration 512.
[2025-11-13 11:24:52,077][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:24:52,078][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:25:00,913][__main__][INFO] - Number of regex retries in iteration 512: 0
[2025-11-13 11:25:00,914][__main__][INFO] - agents played in iteration 512 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:25:01,335][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:01,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:01,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:01,437][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:01,437][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:25:01,438][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:25:02,106][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:25:02,570][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:25:02,893][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:25:03,216][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:25:03,540][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:25:03,864][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:25:04,188][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:25:04,514][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:25:04,839][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:25:05,162][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:25:05,486][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:25:05,814][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:25:06,140][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:25:06,463][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:25:06,786][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:25:07,109][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:25:07,433][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:25:07,756][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:25:08,080][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:25:08,406][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:25:08,731][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:25:09,056][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:25:09,382][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:25:09,707][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:25:10,030][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:25:10,354][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:25:10,678][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:25:11,003][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:25:11,327][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:25:11,648][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:25:11,972][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:25:12,297][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:25:12,621][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:25:13,319][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:25:14,016][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:25:14,018][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:25:14,019][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:25:14,874][__main__][INFO] - Iteration 513 took 22s (38.76% Gen, 57.49% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 41m 27s. Estimated total time: 18h 59m 52s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 59s, 500 more iterations: 3h 9m 58s.
[2025-11-13 11:25:14,876][__main__][INFO] - Starting iteration 513.
[2025-11-13 11:25:14,879][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:25:14,879][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:25:24,145][__main__][INFO] - Number of regex retries in iteration 513: 0
[2025-11-13 11:25:24,145][__main__][INFO] - agents played in iteration 513 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:25:24,572][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:24,605][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:24,638][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:24,672][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:24,672][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:25:24,673][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:25:25,346][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:25:25,640][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:25:25,964][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:25:26,288][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:25:26,613][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:25:26,938][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:25:27,264][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:25:27,590][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:25:27,918][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:25:28,245][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:25:28,570][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:25:28,892][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:25:29,215][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:25:29,542][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:25:29,867][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:25:30,195][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:25:30,520][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:25:30,842][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:25:31,165][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:25:31,488][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:25:31,815][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:25:32,139][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:25:32,465][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:25:32,789][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:25:33,113][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:25:33,438][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:25:33,762][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:25:34,086][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:25:34,410][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:25:34,734][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:25:35,059][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:25:35,381][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:25:35,707][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:25:36,409][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:25:37,110][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:25:37,111][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:25:37,112][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:25:37,978][__main__][INFO] - Iteration 514 took 23s (40.11% Gen, 56.13% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 56m 11s. Estimated total time: 19h 15m 0s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 30s.
[2025-11-13 11:25:37,980][__main__][INFO] - Starting iteration 514.
[2025-11-13 11:25:37,983][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:25:37,984][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:25:47,257][__main__][INFO] - Number of regex retries in iteration 514: 0
[2025-11-13 11:25:47,257][__main__][INFO] - agents played in iteration 514 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:25:47,682][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:47,716][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:47,749][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:47,782][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:25:47,782][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:25:47,783][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:25:48,460][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:25:48,754][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:25:49,078][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:25:49,403][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:25:49,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:25:50,051][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:25:50,380][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:25:50,703][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:25:51,026][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:25:51,350][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:25:51,673][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:25:51,996][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:25:52,319][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:25:52,644][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:25:52,970][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:25:53,294][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:25:53,617][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:25:53,945][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:25:54,274][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:25:54,605][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:25:54,928][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:25:55,252][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:25:55,575][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:25:55,901][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:25:56,226][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:25:56,548][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:25:56,871][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:25:57,196][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:25:57,521][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:25:57,845][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:25:58,169][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:25:58,492][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:25:58,816][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:25:59,527][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:26:00,207][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:26:00,209][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:26:00,211][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:26:01,057][__main__][INFO] - Iteration 515 took 23s (40.19% Gen, 56.14% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 54m 34s. Estimated total time: 19h 13m 46s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 27s, 500 more iterations: 3h 12m 17s.
[2025-11-13 11:26:01,060][__main__][INFO] - Starting iteration 515.
[2025-11-13 11:26:01,062][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:26:01,063][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:26:09,808][__main__][INFO] - Number of regex retries in iteration 515: 0
[2025-11-13 11:26:09,809][__main__][INFO] - agents played in iteration 515 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:26:10,234][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:10,270][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:10,304][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:10,337][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:10,337][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:26:10,338][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:26:11,018][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:26:11,312][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:26:11,634][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:26:11,956][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:26:12,280][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:26:12,609][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:26:12,935][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:26:13,258][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:26:13,581][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:26:13,909][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:26:14,238][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:26:14,563][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:26:14,887][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:26:15,211][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:26:15,532][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:26:15,856][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:26:16,180][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:26:16,505][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:26:16,829][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:26:17,152][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:26:17,479][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:26:17,805][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:26:18,132][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:26:18,456][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:26:18,779][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:26:19,105][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:26:19,428][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:26:19,751][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:26:20,079][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:26:20,404][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:26:20,729][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:26:21,052][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:26:21,376][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:26:22,074][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:26:22,759][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:26:22,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:26:22,761][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:26:23,581][__main__][INFO] - Iteration 516 took 22s (38.84% Gen, 57.52% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 26m 23s. Estimated total time: 18h 45m 58s. Time estimates for 10 more iterations: 3m 45s, 100 more iterations: 37m 31s, 500 more iterations: 3h 7m 39s.
[2025-11-13 11:26:23,583][__main__][INFO] - Starting iteration 516.
[2025-11-13 11:26:23,586][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:26:23,586][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:26:32,612][__main__][INFO] - Number of regex retries in iteration 516: 0
[2025-11-13 11:26:32,613][__main__][INFO] - agents played in iteration 516 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:26:33,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:33,074][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:33,107][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:33,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:33,141][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:26:33,141][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:26:33,827][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:26:34,121][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:26:34,444][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:26:34,767][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:26:35,090][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:26:35,412][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:26:35,734][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:26:36,059][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:26:36,388][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:26:36,711][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:26:37,036][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:26:37,362][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:26:37,689][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:26:38,012][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:26:38,337][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:26:38,660][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:26:38,987][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:26:39,310][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:26:39,634][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:26:39,957][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:26:40,280][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:26:40,606][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:26:40,930][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:26:41,254][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:26:41,578][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:26:41,902][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:26:42,226][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:26:42,551][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:26:42,877][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:26:43,201][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:26:43,526][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:26:43,851][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:26:44,175][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:26:44,876][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:26:45,566][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:26:45,568][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:26:45,570][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:26:46,396][__main__][INFO] - Iteration 517 took 22s (39.57% Gen, 56.80% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 40m 37s. Estimated total time: 19h 0m 34s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 1s, 500 more iterations: 3h 10m 5s.
[2025-11-13 11:26:46,400][__main__][INFO] - Starting iteration 517.
[2025-11-13 11:26:46,403][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:26:46,404][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:26:55,758][__main__][INFO] - Number of regex retries in iteration 517: 0
[2025-11-13 11:26:55,758][__main__][INFO] - agents played in iteration 517 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:26:56,188][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:56,223][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:56,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:56,289][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:26:56,290][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:26:56,290][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:26:56,964][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:26:57,259][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:26:57,583][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:26:57,905][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:26:58,233][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:26:58,559][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:26:58,882][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:26:59,205][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:26:59,530][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:26:59,859][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:27:00,187][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:27:00,513][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:27:00,838][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:27:01,161][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:27:01,488][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:27:01,814][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:27:02,139][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:27:02,461][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:27:02,789][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:27:03,113][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:27:03,438][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:27:03,761][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:27:04,086][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:27:04,411][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:27:04,734][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:27:05,060][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:27:05,385][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:27:05,710][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:27:06,034][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:27:06,358][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:27:06,684][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:27:07,007][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:27:07,331][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:27:08,029][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:27:08,721][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:27:08,722][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:27:08,724][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:27:09,532][__main__][INFO] - Iteration 518 took 23s (40.45% Gen, 56.05% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 56m 8s. Estimated total time: 19h 16m 29s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 32s, 500 more iterations: 3h 12m 44s.
[2025-11-13 11:27:09,534][__main__][INFO] - Starting iteration 518.
[2025-11-13 11:27:09,536][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:27:09,537][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:27:18,594][__main__][INFO] - Number of regex retries in iteration 518: 0
[2025-11-13 11:27:18,595][__main__][INFO] - agents played in iteration 518 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:27:19,031][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:19,064][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:19,097][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:19,131][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:19,131][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:27:19,131][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:27:19,811][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:27:20,105][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:27:20,429][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:27:20,754][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:27:21,080][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:27:21,407][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:27:21,732][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:27:22,055][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:27:22,379][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:27:22,707][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:27:23,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:27:23,362][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:27:23,691][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:27:24,020][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:27:24,344][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:27:24,670][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:27:24,996][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:27:25,320][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:27:25,643][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:27:25,968][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:27:26,292][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:27:26,616][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:27:26,941][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:27:27,266][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:27:27,591][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:27:27,920][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:27:28,244][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:27:28,570][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:27:28,893][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:27:29,218][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:27:29,547][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:27:29,870][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:27:30,194][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:27:30,901][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:27:31,592][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:27:31,593][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:27:31,595][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:27:32,565][__main__][INFO] - Iteration 519 took 23s (39.33% Gen, 56.45% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 50m 46s. Estimated total time: 19h 11m 30s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 55s.
[2025-11-13 11:27:32,568][__main__][INFO] - Starting iteration 519.
[2025-11-13 11:27:32,571][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:27:32,572][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:27:41,767][__main__][INFO] - Number of regex retries in iteration 519: 0
[2025-11-13 11:27:41,768][__main__][INFO] - agents played in iteration 519 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:27:42,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:42,243][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:42,277][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:42,310][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:27:42,311][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:27:42,311][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:27:43,005][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:27:43,301][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:27:43,626][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:27:43,954][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:27:44,275][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:27:44,601][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:27:44,925][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:27:45,249][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:27:45,572][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:27:45,898][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:27:46,226][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:27:46,550][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:27:46,876][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:27:47,199][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:27:47,525][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:27:47,850][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:27:48,175][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:27:48,499][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:27:48,823][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:27:49,149][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:27:49,474][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:27:49,797][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:27:50,120][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:27:50,445][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:27:50,770][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:27:51,094][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:27:51,419][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:27:51,742][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:27:52,069][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:27:52,394][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:27:52,719][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:27:53,045][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:27:53,370][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:27:54,067][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:27:54,752][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:27:54,753][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:27:54,755][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:27:55,619][__main__][INFO] - Iteration 520 took 23s (39.90% Gen, 56.35% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 51m 19s. Estimated total time: 19h 12m 26s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 24s, 500 more iterations: 3h 12m 4s.
[2025-11-13 11:27:55,621][__main__][INFO] - Starting iteration 520.
[2025-11-13 11:27:55,623][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 51 and human policies 1.
[2025-11-13 11:27:55,624][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:28:04,969][__main__][INFO] - Number of regex retries in iteration 520: 0
[2025-11-13 11:28:04,969][__main__][INFO] - agents played in iteration 520 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:28:05,397][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:05,432][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:05,466][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:05,500][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:05,500][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:28:05,500][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:28:06,197][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:28:06,492][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:28:06,815][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:28:07,145][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:28:07,466][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:28:07,798][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:28:08,124][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:28:08,447][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:28:08,773][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:28:09,098][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:28:09,425][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:28:09,749][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:28:10,077][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:28:10,402][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:28:10,727][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:28:11,051][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:28:11,376][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:28:11,699][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:28:12,023][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:28:12,346][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:28:12,670][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:28:12,997][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:28:13,321][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:28:13,644][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:28:13,969][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:28:14,295][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:28:14,620][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:28:14,945][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:28:15,269][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:28:15,593][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:28:15,918][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:28:16,242][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:28:16,568][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:28:17,273][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:28:17,961][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:28:17,962][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:28:17,963][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:28:19,599][__main__][INFO] - Iteration 521 took 23s (38.98% Gen, 54.20% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 37m 18s. Estimated total time: 19h 58m 49s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 57s, 500 more iterations: 3h 19m 48s.
[2025-11-13 11:28:19,600][__main__][INFO] - Starting iteration 521.
[2025-11-13 11:28:19,603][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:28:19,604][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:28:28,545][__main__][INFO] - Number of regex retries in iteration 521: 0
[2025-11-13 11:28:28,546][__main__][INFO] - agents played in iteration 521 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:28:28,969][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:29,005][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:29,040][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:29,073][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:29,073][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:28:29,074][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:28:29,749][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:28:30,044][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:28:30,372][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:28:30,700][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:28:31,024][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:28:31,348][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:28:31,673][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:28:31,996][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:28:32,322][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:28:32,648][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:28:32,973][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:28:33,299][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:28:33,625][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:28:33,949][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:28:34,273][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:28:34,596][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:28:34,920][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:28:35,245][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:28:35,570][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:28:35,894][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:28:36,220][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:28:36,547][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:28:36,872][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:28:37,197][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:28:37,521][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:28:37,846][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:28:38,171][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:28:38,494][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:28:38,818][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:28:39,142][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:28:39,467][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:28:39,794][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:28:40,119][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:28:40,827][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:28:41,510][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:28:41,512][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:28:41,513][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:28:42,317][__main__][INFO] - Iteration 522 took 22s (39.36% Gen, 57.09% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 33m 51s. Estimated total time: 18h 55m 44s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 51s, 500 more iterations: 3h 9m 17s.
[2025-11-13 11:28:42,319][__main__][INFO] - Starting iteration 522.
[2025-11-13 11:28:42,322][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:28:42,323][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:28:52,044][__main__][INFO] - Number of regex retries in iteration 522: 0
[2025-11-13 11:28:52,045][__main__][INFO] - agents played in iteration 522 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:28:52,467][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:52,500][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:52,532][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:52,565][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:28:52,566][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:28:52,566][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:28:53,262][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:28:53,556][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:28:53,881][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:28:54,203][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:28:54,527][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:28:54,851][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:28:55,174][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:28:55,500][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:28:55,826][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:28:56,152][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:28:56,477][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:28:56,802][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:28:57,129][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:28:57,451][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:28:57,775][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:28:58,100][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:28:58,424][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:28:58,748][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:28:59,073][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:28:59,396][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:28:59,720][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:29:00,045][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:29:00,372][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:29:00,696][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:29:01,020][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:29:01,345][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:29:01,670][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:29:01,995][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:29:02,320][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:29:02,644][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:29:02,969][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:29:03,295][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:29:03,619][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:29:04,320][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:29:05,011][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:29:05,012][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:29:05,014][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:29:05,855][__main__][INFO] - Iteration 523 took 23s (41.30% Gen, 55.11% Train). Generation: 9s, Training: 12s. Estimated remaining time: 19h 14m 25s. Estimated total time: 19h 36m 41s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 13s, 500 more iterations: 3h 16m 6s.
[2025-11-13 11:29:05,857][__main__][INFO] - Starting iteration 523.
[2025-11-13 11:29:05,860][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:29:05,861][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:29:14,496][__main__][INFO] - Number of regex retries in iteration 523: 0
[2025-11-13 11:29:14,497][__main__][INFO] - agents played in iteration 523 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:29:14,920][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:14,953][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:14,986][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:15,019][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:15,020][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:29:15,020][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:29:15,698][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:29:15,993][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:29:16,316][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:29:16,639][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:29:16,962][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:29:17,288][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:29:17,615][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:29:17,938][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:29:18,263][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:29:18,589][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:29:18,914][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:29:19,243][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:29:19,567][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:29:19,892][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:29:20,217][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:29:20,541][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:29:20,866][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:29:21,190][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:29:21,515][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:29:21,839][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:29:22,165][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:29:22,488][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:29:22,812][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:29:23,139][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:29:23,463][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:29:23,788][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:29:24,113][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:29:24,438][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:29:24,764][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:29:25,089][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:29:25,413][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:29:25,738][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:29:26,062][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:29:26,765][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:29:27,459][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:29:27,460][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:29:27,461][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:29:28,328][__main__][INFO] - Iteration 524 took 22s (38.44% Gen, 57.70% Train). Generation: 8s, Training: 12s. Estimated remaining time: 18h 20m 46s. Estimated total time: 18h 43m 25s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 26s, 500 more iterations: 3h 7m 14s.
[2025-11-13 11:29:28,330][__main__][INFO] - Starting iteration 524.
[2025-11-13 11:29:28,333][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:29:28,333][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:29:37,648][__main__][INFO] - Number of regex retries in iteration 524: 0
[2025-11-13 11:29:37,649][__main__][INFO] - agents played in iteration 524 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:29:38,081][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:38,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:38,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:38,181][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:29:38,182][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:29:38,182][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:29:38,875][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:29:39,169][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:29:39,493][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:29:39,818][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:29:40,142][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:29:40,467][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:29:40,793][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:29:41,117][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:29:41,445][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:29:41,770][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:29:42,094][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:29:42,418][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:29:42,743][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:29:43,068][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:29:43,393][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:29:43,718][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:29:44,042][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:29:44,364][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:29:44,688][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:29:45,012][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:29:45,338][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:29:45,663][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:29:45,987][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:29:46,315][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:29:46,645][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:29:46,974][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:29:47,301][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:29:47,625][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:29:47,952][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:29:48,276][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:29:48,603][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:29:48,926][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:29:49,252][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:29:49,962][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:29:50,647][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:29:50,650][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:29:50,652][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:29:51,488][__main__][INFO] - Iteration 525 took 23s (40.24% Gen, 56.15% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 54m 45s. Estimated total time: 19h 17m 48s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 35s, 500 more iterations: 3h 12m 58s.
[2025-11-13 11:29:51,490][__main__][INFO] - Starting iteration 525.
[2025-11-13 11:29:51,493][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:29:51,494][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:30:00,601][__main__][INFO] - Number of regex retries in iteration 525: 0
[2025-11-13 11:30:00,602][__main__][INFO] - agents played in iteration 525 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:30:01,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:01,068][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:01,101][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:01,134][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:01,134][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:30:01,135][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:30:01,844][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:30:02,139][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:30:02,465][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:30:02,789][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:30:03,116][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:30:03,439][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:30:03,765][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:30:04,089][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:30:04,414][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:30:04,739][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:30:05,063][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:30:05,387][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:30:05,714][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:30:06,038][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:30:06,364][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:30:06,689][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:30:07,015][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:30:07,339][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:30:07,664][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:30:07,988][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:30:08,314][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:30:08,636][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:30:08,960][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:30:09,283][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:30:09,610][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:30:09,935][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:30:10,259][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:30:10,586][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:30:10,911][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:30:11,236][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:30:11,560][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:30:11,888][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:30:12,213][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:30:12,928][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:30:13,615][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:30:13,616][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:30:13,618][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:30:14,661][__main__][INFO] - Iteration 526 took 23s (39.31% Gen, 56.17% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 55m 2s. Estimated total time: 19h 18m 27s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 36s, 500 more iterations: 3h 13m 4s.
[2025-11-13 11:30:14,663][__main__][INFO] - Starting iteration 526.
[2025-11-13 11:30:14,666][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:30:14,666][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:30:23,784][__main__][INFO] - Number of regex retries in iteration 526: 0
[2025-11-13 11:30:23,785][__main__][INFO] - agents played in iteration 526 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:30:24,216][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:24,249][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:24,283][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:24,317][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:24,317][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:30:24,318][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:30:25,033][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:30:25,329][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:30:25,654][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:30:25,983][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:30:26,305][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:30:26,629][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:30:26,955][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:30:27,280][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:30:27,603][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:30:27,927][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:30:28,250][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:30:28,573][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:30:28,898][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:30:29,222][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:30:29,546][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:30:29,870][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:30:30,195][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:30:30,520][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:30:30,843][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:30:31,167][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:30:31,490][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:30:31,815][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:30:32,139][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:30:32,466][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:30:32,793][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:30:33,117][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:30:33,446][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:30:33,776][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:30:34,104][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:30:34,428][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:30:34,751][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:30:35,075][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:30:35,400][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:30:36,105][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:30:36,797][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:30:36,798][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:30:36,799][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:30:37,668][__main__][INFO] - Iteration 527 took 23s (39.64% Gen, 56.58% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 46m 20s. Estimated total time: 19h 10m 9s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 20s, 500 more iterations: 3h 11m 41s.
[2025-11-13 11:30:37,671][__main__][INFO] - Starting iteration 527.
[2025-11-13 11:30:37,674][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:30:37,674][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:30:46,813][__main__][INFO] - Number of regex retries in iteration 527: 0
[2025-11-13 11:30:46,813][__main__][INFO] - agents played in iteration 527 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:30:47,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:47,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:47,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:47,363][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:30:47,364][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:30:47,364][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:30:48,075][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:30:48,368][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:30:48,694][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:30:49,022][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:30:49,349][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:30:49,673][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:30:49,996][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:30:50,322][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:30:50,647][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:30:50,971][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:30:51,296][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:30:51,620][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:30:51,947][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:30:52,274][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:30:52,597][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:30:52,919][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:30:53,243][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:30:53,567][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:30:53,890][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:30:54,214][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:30:54,538][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:30:54,862][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:30:55,187][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:30:55,511][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:30:55,838][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:30:56,163][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:30:56,488][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:30:56,817][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:30:57,144][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:30:57,470][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:30:57,794][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:30:58,117][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:30:58,443][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:30:59,158][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:30:59,846][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:30:59,848][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:30:59,849][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:31:00,663][__main__][INFO] - Iteration 528 took 22s (39.75% Gen, 56.70% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 45m 18s. Estimated total time: 19h 9m 30s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 19s, 500 more iterations: 3h 11m 35s.
[2025-11-13 11:31:00,665][__main__][INFO] - Starting iteration 528.
[2025-11-13 11:31:00,668][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:31:00,668][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:31:09,814][__main__][INFO] - Number of regex retries in iteration 528: 0
[2025-11-13 11:31:09,815][__main__][INFO] - agents played in iteration 528 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:31:10,256][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:10,290][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:10,326][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:10,360][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:10,360][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:31:10,361][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:31:11,092][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:31:11,389][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:31:11,716][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:31:12,040][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:31:12,365][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:31:12,689][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:31:13,013][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:31:13,337][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:31:13,660][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:31:13,985][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:31:14,311][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:31:14,635][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:31:14,960][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:31:15,285][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:31:15,610][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:31:15,934][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:31:16,259][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:31:16,585][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:31:16,911][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:31:17,237][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:31:17,561][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:31:17,886][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:31:18,213][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:31:18,536][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:31:18,861][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:31:19,186][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:31:19,512][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:31:19,836][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:31:20,159][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:31:20,482][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:31:20,807][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:31:21,135][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:31:21,460][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:31:22,187][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:31:22,887][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:31:22,889][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:31:22,890][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:31:23,800][__main__][INFO] - Iteration 529 took 23s (39.54% Gen, 56.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 52m 3s. Estimated total time: 19h 16m 38s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 33s, 500 more iterations: 3h 12m 46s.
[2025-11-13 11:31:23,801][__main__][INFO] - Starting iteration 529.
[2025-11-13 11:31:23,804][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:31:23,804][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:31:32,763][__main__][INFO] - Number of regex retries in iteration 529: 0
[2025-11-13 11:31:32,763][__main__][INFO] - agents played in iteration 529 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:31:33,205][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:33,239][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:33,273][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:33,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:33,308][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:31:33,308][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:31:34,020][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:31:34,316][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:31:34,639][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:31:34,965][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:31:35,290][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:31:35,615][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:31:35,941][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:31:36,265][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:31:36,590][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:31:36,915][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:31:37,239][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:31:37,562][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:31:37,887][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:31:38,212][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:31:38,538][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:31:38,863][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:31:39,187][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:31:39,512][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:31:39,837][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:31:40,160][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:31:40,487][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:31:40,812][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:31:41,137][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:31:41,461][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:31:41,790][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:31:42,114][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:31:42,438][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:31:42,764][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:31:43,089][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:31:43,414][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:31:43,739][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:31:44,063][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:31:44,388][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:31:45,099][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:31:45,785][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:31:45,787][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:31:45,790][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:31:46,628][__main__][INFO] - Iteration 530 took 22s (39.25% Gen, 57.07% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 36m 17s. Estimated total time: 19h 1m 14s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 2s, 500 more iterations: 3h 10m 12s.
[2025-11-13 11:31:46,630][__main__][INFO] - Starting iteration 530.
[2025-11-13 11:31:46,633][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 52 and human policies 1.
[2025-11-13 11:31:46,633][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:31:55,691][__main__][INFO] - Number of regex retries in iteration 530: 0
[2025-11-13 11:31:55,691][__main__][INFO] - agents played in iteration 530 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:31:56,130][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:56,164][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:56,199][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:56,232][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:31:56,232][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:31:56,233][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:31:56,962][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:31:57,257][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:31:57,582][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:31:57,911][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:31:58,238][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:31:58,565][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:31:58,889][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:31:59,214][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:31:59,539][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:31:59,866][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:32:00,189][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:32:00,515][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:32:00,839][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:32:01,166][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:32:01,492][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:32:01,816][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:32:02,140][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:32:02,466][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:32:02,790][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:32:03,113][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:32:03,437][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:32:03,766][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:32:04,091][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:32:04,417][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:32:04,747][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:32:05,074][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:32:05,405][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:32:05,734][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:32:06,060][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:32:06,384][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:32:06,715][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:32:07,042][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:32:07,367][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:32:08,068][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:32:08,759][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:32:08,760][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:32:08,762][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:32:10,337][__main__][INFO] - Iteration 531 took 23s (38.21% Gen, 55.14% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 19m 55s. Estimated total time: 19h 45m 17s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 32s.
[2025-11-13 11:32:10,339][__main__][INFO] - Starting iteration 531.
[2025-11-13 11:32:10,342][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:32:10,343][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:32:19,313][__main__][INFO] - Number of regex retries in iteration 531: 0
[2025-11-13 11:32:19,314][__main__][INFO] - agents played in iteration 531 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:32:19,748][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:19,782][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:19,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:19,852][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:19,852][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:32:19,853][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:32:20,571][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:32:20,866][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:32:21,190][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:32:21,513][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:32:21,838][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:32:22,163][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:32:22,487][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:32:22,811][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:32:23,134][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:32:23,459][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:32:23,784][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:32:24,110][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:32:24,436][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:32:24,759][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:32:25,085][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:32:25,412][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:32:25,736][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:32:26,060][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:32:26,386][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:32:26,710][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:32:27,038][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:32:27,364][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:32:27,689][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:32:28,013][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:32:28,338][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:32:28,663][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:32:28,988][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:32:29,312][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:32:29,636][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:32:29,960][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:32:30,287][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:32:30,615][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:32:30,939][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:32:31,634][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:32:32,324][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:32:32,326][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:32:32,327][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:32:33,142][__main__][INFO] - Iteration 532 took 22s (39.34% Gen, 57.07% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 34m 19s. Estimated total time: 19h 0m 3s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 0s, 500 more iterations: 3h 10m 0s.
[2025-11-13 11:32:33,144][__main__][INFO] - Starting iteration 532.
[2025-11-13 11:32:33,148][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:32:33,149][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:32:41,258][__main__][INFO] - Number of regex retries in iteration 532: 0
[2025-11-13 11:32:41,259][__main__][INFO] - agents played in iteration 532 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:32:41,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:41,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:41,768][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:41,802][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:32:41,803][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:32:41,804][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:32:42,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:32:42,813][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:32:43,138][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:32:43,462][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:32:43,786][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:32:44,112][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:32:44,437][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:32:44,760][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:32:45,085][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:32:45,409][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:32:45,734][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:32:46,058][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:32:46,382][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:32:46,706][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:32:47,030][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:32:47,355][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:32:47,680][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:32:48,005][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:32:48,330][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:32:48,654][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:32:48,977][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:32:49,301][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:32:49,625][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:32:49,949][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:32:50,273][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:32:50,598][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:32:50,928][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:32:51,255][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:32:51,581][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:32:51,906][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:32:52,229][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:32:52,554][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:32:52,879][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:32:53,598][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:32:54,288][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:32:54,289][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:32:54,290][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:32:55,150][__main__][INFO] - Iteration 533 took 22s (36.85% Gen, 59.23% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 54m 2s. Estimated total time: 18h 20m 8s. Time estimates for 10 more iterations: 3m 40s, 100 more iterations: 36m 40s, 500 more iterations: 3h 3m 21s.
[2025-11-13 11:32:55,152][__main__][INFO] - Starting iteration 533.
[2025-11-13 11:32:55,156][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:32:55,156][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:33:03,504][__main__][INFO] - Number of regex retries in iteration 533: 0
[2025-11-13 11:33:03,505][__main__][INFO] - agents played in iteration 533 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:33:03,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:03,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:04,009][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:04,043][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:04,044][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:33:04,044][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:33:04,761][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:33:05,055][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:33:05,381][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:33:05,707][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:33:06,032][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:33:06,356][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:33:06,683][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:33:07,007][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:33:07,332][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:33:07,657][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:33:07,982][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:33:08,309][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:33:08,634][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:33:08,957][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:33:09,281][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:33:09,606][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:33:09,931][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:33:10,259][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:33:10,585][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:33:10,910][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:33:11,236][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:33:11,560][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:33:11,886][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:33:12,212][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:33:12,538][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:33:12,864][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:33:13,188][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:33:13,512][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:33:13,838][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:33:14,161][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:33:14,490][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:33:14,819][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:33:15,145][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:33:15,853][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:33:16,569][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:33:16,571][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:33:16,574][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:33:17,377][__main__][INFO] - Iteration 534 took 22s (37.57% Gen, 58.81% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 4m 38s. Estimated total time: 18h 31m 7s. Time estimates for 10 more iterations: 3m 42s, 100 more iterations: 37m 2s, 500 more iterations: 3h 5m 11s.
[2025-11-13 11:33:17,379][__main__][INFO] - Starting iteration 534.
[2025-11-13 11:33:17,382][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:33:17,383][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:33:26,492][__main__][INFO] - Number of regex retries in iteration 534: 0
[2025-11-13 11:33:26,493][__main__][INFO] - agents played in iteration 534 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:33:26,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:26,967][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:27,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:27,035][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:27,036][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:33:27,036][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:33:27,762][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:33:28,058][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:33:28,387][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:33:28,712][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:33:29,034][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:33:29,359][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:33:29,684][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:33:30,009][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:33:30,333][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:33:30,656][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:33:30,980][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:33:31,305][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:33:31,632][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:33:31,954][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:33:32,278][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:33:32,602][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:33:32,927][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:33:33,253][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:33:33,578][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:33:33,903][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:33:34,228][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:33:34,553][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:33:34,877][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:33:35,208][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:33:35,535][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:33:35,862][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:33:36,186][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:33:36,509][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:33:36,833][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:33:37,159][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:33:37,484][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:33:37,807][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:33:38,133][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:33:38,842][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:33:39,570][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:33:39,571][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:33:39,573][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:33:40,428][__main__][INFO] - Iteration 535 took 23s (39.53% Gen, 56.75% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 45m 28s. Estimated total time: 19h 12m 19s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 24s, 500 more iterations: 3h 12m 3s.
[2025-11-13 11:33:40,430][__main__][INFO] - Starting iteration 535.
[2025-11-13 11:33:40,433][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:33:40,434][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:33:49,970][__main__][INFO] - Number of regex retries in iteration 535: 0
[2025-11-13 11:33:49,970][__main__][INFO] - agents played in iteration 535 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:33:50,411][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,444][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,478][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,513][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:33:50,513][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:33:50,513][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:33:51,245][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:33:51,541][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:33:51,866][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:33:52,191][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:33:52,514][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:33:52,838][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:33:53,163][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:33:53,486][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:33:53,810][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:33:54,134][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:33:54,460][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:33:54,783][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:33:55,107][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:33:55,431][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:33:55,757][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:33:56,080][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:33:56,405][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:33:56,730][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:33:57,056][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:33:57,381][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:33:57,706][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:33:58,030][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:33:58,359][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:33:58,682][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:33:59,007][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:33:59,332][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:33:59,655][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:33:59,978][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:34:00,305][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:34:00,631][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:34:00,955][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:34:01,280][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:34:01,606][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:34:02,304][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:34:03,019][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:34:03,020][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:34:03,022][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:34:04,009][__main__][INFO] - Iteration 536 took 23s (40.45% Gen, 55.36% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 11m 35s. Estimated total time: 19h 38m 50s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 17s, 500 more iterations: 3h 16m 28s.
[2025-11-13 11:34:04,012][__main__][INFO] - Starting iteration 536.
[2025-11-13 11:34:04,015][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:34:04,015][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:34:13,300][__main__][INFO] - Number of regex retries in iteration 536: 0
[2025-11-13 11:34:13,301][__main__][INFO] - agents played in iteration 536 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:34:13,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,811][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,845][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:13,846][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:34:13,846][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:34:14,570][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:34:14,866][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:34:15,192][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:34:15,515][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:34:15,838][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:34:16,163][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:34:16,488][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:34:16,813][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:34:17,138][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:34:17,464][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:34:17,788][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:34:18,113][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:34:18,437][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:34:18,763][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:34:19,089][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:34:19,413][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:34:19,738][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:34:20,062][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:34:20,386][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:34:20,712][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:34:21,042][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:34:21,368][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:34:21,695][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:34:22,018][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:34:22,345][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:34:22,672][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:34:23,003][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:34:23,331][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:34:23,656][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:34:23,982][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:34:24,308][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:34:24,633][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:34:24,956][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:34:25,660][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:34:26,378][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:34:26,380][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:34:26,382][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:34:27,338][__main__][INFO] - Iteration 537 took 23s (39.82% Gen, 56.08% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 58m 33s. Estimated total time: 19h 26m 11s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 52s, 500 more iterations: 3h 14m 21s.
[2025-11-13 11:34:27,340][__main__][INFO] - Starting iteration 537.
[2025-11-13 11:34:27,343][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:34:27,344][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:34:36,052][__main__][INFO] - Number of regex retries in iteration 537: 0
[2025-11-13 11:34:36,053][__main__][INFO] - agents played in iteration 537 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:34:36,508][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,543][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,577][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,611][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:36,612][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:34:36,612][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:34:37,347][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:34:37,643][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:34:37,968][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:34:38,293][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:34:38,616][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:34:38,940][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:34:39,265][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:34:39,591][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:34:39,914][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:34:40,238][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:34:40,562][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:34:40,887][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:34:41,212][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:34:41,537][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:34:41,862][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:34:42,187][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:34:42,512][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:34:42,836][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:34:43,166][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:34:43,497][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:34:43,823][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:34:44,148][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:34:44,477][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:34:44,802][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:34:45,129][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:34:45,458][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:34:45,787][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:34:46,111][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:34:46,439][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:34:46,765][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:34:47,094][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:34:47,417][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:34:47,739][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:34:48,442][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:34:49,165][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:34:49,166][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:34:49,168][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:34:50,104][__main__][INFO] - Iteration 538 took 22s (38.26% Gen, 57.62% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 30m 3s. Estimated total time: 18h 58m 4s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 56s, 500 more iterations: 3h 9m 40s.
[2025-11-13 11:34:50,106][__main__][INFO] - Starting iteration 538.
[2025-11-13 11:34:50,109][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:34:50,110][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:34:57,763][mllm.models.large_language_model_local][WARNING] - Response *Aạch shoutout to the other agent for the consistent play so far. I'll keep it going.*

<A> did not match regex: (<A>|<B>), retry 1/1
[2025-11-13 11:34:59,383][__main__][INFO] - Number of regex retries in iteration 538: 1
[2025-11-13 11:34:59,383][__main__][INFO] - agents played in iteration 538 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:34:59,829][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,930][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:34:59,930][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:34:59,930][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:35:00,670][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:35:00,965][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:35:01,289][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:35:01,613][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:35:01,938][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:35:02,262][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:35:02,586][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:35:02,911][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:35:03,236][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:35:03,560][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:35:03,884][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:35:04,210][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:35:04,541][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:35:04,867][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:35:05,192][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:35:05,516][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:35:05,841][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:35:06,164][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:35:06,488][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:35:06,814][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:35:07,138][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:35:07,464][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:35:07,788][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:35:08,112][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:35:08,438][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:35:08,767][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:35:09,097][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:35:09,428][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:35:09,751][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:35:10,074][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:35:10,401][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:35:10,725][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:35:11,048][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:35:11,746][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:35:12,463][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:35:12,464][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:35:12,466][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:35:13,353][__main__][INFO] - Iteration 539 took 23s (39.90% Gen, 56.28% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 53m 50s. Estimated total time: 19h 22m 14s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 44s, 500 more iterations: 3h 13m 42s.
[2025-11-13 11:35:13,356][__main__][INFO] - Starting iteration 539.
[2025-11-13 11:35:13,359][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:35:13,359][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:35:22,639][__main__][INFO] - Number of regex retries in iteration 539: 0
[2025-11-13 11:35:22,640][__main__][INFO] - agents played in iteration 539 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:35:23,104][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,138][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,172][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,207][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:23,207][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:35:23,208][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:35:23,935][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:35:24,232][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:35:24,557][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:35:24,881][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:35:25,207][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:35:25,531][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:35:25,854][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:35:26,178][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:35:26,502][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:35:26,827][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:35:27,151][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:35:27,476][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:35:27,800][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:35:28,123][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:35:28,445][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:35:28,768][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:35:29,093][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:35:29,418][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:35:29,741][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:35:30,066][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:35:30,388][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:35:30,712][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:35:31,039][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:35:31,364][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:35:31,688][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:35:32,014][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:35:32,339][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:35:32,663][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:35:32,986][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:35:33,312][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:35:33,637][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:35:33,959][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:35:34,285][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:35:34,976][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:35:35,693][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:35:35,694][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:35:35,696][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:35:36,595][__main__][INFO] - Iteration 540 took 23s (39.94% Gen, 56.19% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 53m 4s. Estimated total time: 19h 21m 51s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 38s.
[2025-11-13 11:35:36,597][__main__][INFO] - Starting iteration 540.
[2025-11-13 11:35:36,600][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 53 and human policies 1.
[2025-11-13 11:35:36,601][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:35:46,277][__main__][INFO] - Number of regex retries in iteration 540: 0
[2025-11-13 11:35:46,278][__main__][INFO] - agents played in iteration 540 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:35:46,737][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,771][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,805][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,838][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:35:46,839][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:35:46,839][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:35:47,557][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:35:47,851][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:35:48,176][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:35:48,500][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:35:48,825][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:35:49,148][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:35:49,473][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:35:49,796][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:35:50,119][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:35:50,442][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:35:50,768][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:35:51,093][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:35:51,417][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:35:51,740][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:35:52,064][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:35:52,390][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:35:52,718][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:35:53,043][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:35:53,367][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:35:53,693][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:35:54,024][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:35:54,349][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:35:54,672][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:35:54,998][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:35:55,323][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:35:55,645][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:35:55,970][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:35:56,293][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:35:56,619][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:35:56,942][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:35:57,266][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:35:57,594][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:35:57,919][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:35:58,605][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:35:59,364][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:35:59,365][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:35:59,367][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:36:00,982][__main__][INFO] - Iteration 541 took 24s (39.69% Gen, 53.68% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 49m 55s. Estimated total time: 20h 19m 7s. Time estimates for 10 more iterations: 4m 3s, 100 more iterations: 40m 38s, 500 more iterations: 3h 23m 11s.
[2025-11-13 11:36:00,984][__main__][INFO] - Starting iteration 541.
[2025-11-13 11:36:00,986][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:36:00,987][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:36:10,619][__main__][INFO] - Number of regex retries in iteration 541: 0
[2025-11-13 11:36:10,619][__main__][INFO] - agents played in iteration 541 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:36:11,059][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:11,093][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:11,126][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:11,160][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:11,161][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:36:11,162][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:36:11,871][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:36:12,165][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:36:12,489][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:36:12,814][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:36:13,139][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:36:13,462][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:36:13,787][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:36:14,112][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:36:14,436][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:36:14,761][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:36:15,087][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:36:15,411][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:36:15,735][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:36:16,058][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:36:16,381][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:36:16,707][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:36:17,036][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:36:17,361][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:36:17,686][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:36:18,012][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:36:18,337][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:36:18,662][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:36:18,988][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:36:19,313][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:36:19,636][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:36:19,959][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:36:20,283][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:36:20,608][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:36:20,931][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:36:21,254][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:36:21,581][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:36:21,912][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:36:22,242][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:36:22,931][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:36:23,641][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:36:23,642][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:36:23,644][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:36:24,490][__main__][INFO] - Iteration 542 took 23s (40.98% Gen, 55.42% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 5m 37s. Estimated total time: 19h 35m 12s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 10s, 500 more iterations: 3h 15m 52s.
[2025-11-13 11:36:24,492][__main__][INFO] - Starting iteration 542.
[2025-11-13 11:36:24,495][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:36:24,495][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:36:33,733][__main__][INFO] - Number of regex retries in iteration 542: 0
[2025-11-13 11:36:33,734][__main__][INFO] - agents played in iteration 542 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:36:34,175][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:34,208][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:34,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:34,276][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:34,276][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:36:34,278][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:36:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:36:35,282][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:36:35,608][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:36:35,932][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:36:36,256][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:36:36,579][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:36:36,906][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:36:37,230][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:36:37,553][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:36:37,876][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:36:38,203][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:36:38,527][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:36:38,852][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:36:39,177][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:36:39,502][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:36:39,830][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:36:40,158][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:36:40,488][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:36:40,813][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:36:41,137][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:36:41,461][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:36:41,786][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:36:42,110][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:36:42,434][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:36:42,758][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:36:43,085][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:36:43,409][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:36:43,733][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:36:44,059][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:36:44,387][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:36:44,711][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:36:45,036][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:36:45,364][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:36:46,065][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:36:46,752][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:36:46,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:36:46,755][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:36:47,548][__main__][INFO] - Iteration 543 took 23s (40.07% Gen, 56.48% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 42m 43s. Estimated total time: 19h 12m 41s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 25s, 500 more iterations: 3h 12m 6s.
[2025-11-13 11:36:47,550][__main__][INFO] - Starting iteration 543.
[2025-11-13 11:36:47,552][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:36:47,553][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:36:57,203][__main__][INFO] - Number of regex retries in iteration 543: 0
[2025-11-13 11:36:57,204][__main__][INFO] - agents played in iteration 543 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:36:57,656][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:57,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:57,723][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:57,757][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:36:57,758][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:36:57,758][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:36:58,469][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:36:58,764][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:36:59,089][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:36:59,412][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:36:59,735][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:37:00,058][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:37:00,382][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:37:00,706][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:37:01,034][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:37:01,359][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:37:01,682][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:37:02,007][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:37:02,330][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:37:02,657][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:37:02,986][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:37:03,310][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:37:03,638][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:37:03,965][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:37:04,292][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:37:04,619][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:37:04,943][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:37:05,268][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:37:05,593][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:37:05,919][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:37:06,243][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:37:06,572][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:37:06,897][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:37:07,228][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:37:07,552][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:37:07,880][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:37:08,209][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:37:08,533][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:37:08,858][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:37:09,557][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:37:10,298][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:37:10,299][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:37:10,301][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:37:11,100][__main__][INFO] - Iteration 544 took 23s (40.98% Gen, 55.62% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 7m 3s. Estimated total time: 19h 37m 25s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 14s, 500 more iterations: 3h 16m 14s.
[2025-11-13 11:37:11,102][__main__][INFO] - Starting iteration 544.
[2025-11-13 11:37:11,104][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:37:11,105][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:37:20,148][__main__][INFO] - Number of regex retries in iteration 544: 0
[2025-11-13 11:37:20,148][__main__][INFO] - agents played in iteration 544 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:37:20,598][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,632][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,666][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,700][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:20,700][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:37:20,701][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:37:21,426][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:37:21,721][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:37:22,045][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:37:22,369][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:37:22,692][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:37:23,017][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:37:23,340][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:37:23,664][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:37:23,990][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:37:24,315][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:37:24,640][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:37:24,964][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:37:25,292][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:37:25,618][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:37:25,943][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:37:26,269][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:37:26,599][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:37:26,922][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:37:27,246][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:37:27,570][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:37:27,899][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:37:28,227][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:37:28,554][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:37:28,877][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:37:29,201][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:37:29,527][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:37:29,852][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:37:30,178][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:37:30,505][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:37:30,831][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:37:31,156][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:37:31,479][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:37:31,806][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:37:32,500][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:37:33,216][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:37:33,217][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:37:33,219][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:37:34,089][__main__][INFO] - Iteration 545 took 22s (39.34% Gen, 56.86% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 38m 32s. Estimated total time: 19h 9m 17s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 18s, 500 more iterations: 3h 11m 32s.
[2025-11-13 11:37:34,091][__main__][INFO] - Starting iteration 545.
[2025-11-13 11:37:34,094][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:37:34,095][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:37:43,152][__main__][INFO] - Number of regex retries in iteration 545: 0
[2025-11-13 11:37:43,152][__main__][INFO] - agents played in iteration 545 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:37:43,606][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:43,640][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:43,674][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:43,707][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:37:43,708][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:37:43,708][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:37:44,432][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:37:44,730][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:37:45,054][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:37:45,378][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:37:45,701][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:37:46,027][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:37:46,353][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:37:46,677][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:37:47,002][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:37:47,328][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:37:47,651][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:37:47,975][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:37:48,299][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:37:48,623][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:37:48,946][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:37:49,272][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:37:49,595][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:37:49,918][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:37:50,241][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:37:50,567][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:37:50,890][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:37:51,217][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:37:51,547][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:37:51,872][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:37:52,197][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:37:52,521][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:37:52,846][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:37:53,170][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:37:53,494][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:37:53,819][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:37:54,143][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:37:54,466][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:37:54,790][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:37:55,458][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:37:56,198][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:37:56,199][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:37:56,201][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:37:57,103][__main__][INFO] - Iteration 546 took 23s (39.36% Gen, 56.71% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 39m 20s. Estimated total time: 19h 10m 28s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 20s, 500 more iterations: 3h 11m 44s.
[2025-11-13 11:37:57,104][__main__][INFO] - Starting iteration 546.
[2025-11-13 11:37:57,107][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:37:57,108][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:38:06,485][__main__][INFO] - Number of regex retries in iteration 546: 0
[2025-11-13 11:38:06,485][__main__][INFO] - agents played in iteration 546 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:38:06,927][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:06,960][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:06,994][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:07,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:07,030][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:38:07,030][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:38:07,753][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:38:08,049][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:38:08,375][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:38:08,697][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:38:09,023][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:38:09,348][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:38:09,674][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:38:09,999][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:38:10,324][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:38:10,651][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:38:10,975][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:38:11,300][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:38:11,631][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:38:11,957][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:38:12,286][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:38:12,612][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:38:12,939][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:38:13,266][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:38:13,593][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:38:13,917][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:38:14,243][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:38:14,569][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:38:14,900][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:38:15,228][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:38:15,555][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:38:15,878][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:38:16,204][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:38:16,530][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:38:16,854][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:38:17,177][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:38:17,503][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:38:17,824][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:38:18,151][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:38:18,829][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:38:19,557][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:38:19,558][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:38:19,560][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:38:20,428][__main__][INFO] - Iteration 547 took 23s (40.21% Gen, 56.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 54m 33s. Estimated total time: 19h 26m 4s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 52s, 500 more iterations: 3h 14m 20s.
[2025-11-13 11:38:20,430][__main__][INFO] - Starting iteration 547.
[2025-11-13 11:38:20,434][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:38:20,435][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:38:29,517][__main__][INFO] - Number of regex retries in iteration 547: 0
[2025-11-13 11:38:29,517][__main__][INFO] - agents played in iteration 547 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:38:29,966][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,000][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:30,068][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:38:30,068][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:38:30,792][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:38:31,089][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:38:31,414][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:38:31,739][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:38:32,064][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:38:32,389][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:38:32,713][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:38:33,037][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:38:33,362][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:38:33,688][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:38:34,012][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:38:34,335][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:38:34,659][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:38:34,986][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:38:35,311][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:38:35,634][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:38:35,958][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:38:36,284][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:38:36,609][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:38:36,933][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:38:37,260][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:38:37,587][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:38:37,913][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:38:38,236][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:38:38,559][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:38:38,882][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:38:39,207][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:38:39,530][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:38:39,855][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:38:40,178][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:38:40,501][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:38:40,825][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:38:41,150][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:38:41,807][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:38:42,546][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:38:42,547][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:38:42,549][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:38:43,464][__main__][INFO] - Iteration 548 took 23s (39.44% Gen, 56.58% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 39m 38s. Estimated total time: 19h 11m 33s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 55s.
[2025-11-13 11:38:43,466][__main__][INFO] - Starting iteration 548.
[2025-11-13 11:38:43,470][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:38:43,470][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:38:53,085][__main__][INFO] - Number of regex retries in iteration 548: 0
[2025-11-13 11:38:53,086][__main__][INFO] - agents played in iteration 548 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:38:53,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,607][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,641][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:38:53,642][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:38:53,642][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:38:54,366][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:38:54,662][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:38:54,987][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:38:55,312][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:38:55,639][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:38:55,964][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:38:56,288][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:38:56,613][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:38:56,938][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:38:57,260][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:38:57,591][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:38:57,920][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:38:58,247][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:38:58,573][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:38:58,899][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:38:59,226][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:38:59,550][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:38:59,876][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:39:00,206][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:39:00,529][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:39:00,855][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:39:01,181][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:39:01,507][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:39:01,832][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:39:02,156][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:39:02,478][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:39:02,804][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:39:03,127][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:39:03,452][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:39:03,776][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:39:04,101][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:39:04,425][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:39:04,750][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:39:05,452][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:39:06,208][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:39:06,210][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:39:06,211][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:39:07,095][__main__][INFO] - Iteration 549 took 23s (40.70% Gen, 55.56% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 8m 59s. Estimated total time: 19h 41m 17s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 22s, 500 more iterations: 3h 16m 52s.
[2025-11-13 11:39:07,096][__main__][INFO] - Starting iteration 549.
[2025-11-13 11:39:07,100][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:39:07,101][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:39:16,337][__main__][INFO] - Number of regex retries in iteration 549: 0
[2025-11-13 11:39:16,338][__main__][INFO] - agents played in iteration 549 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:39:16,795][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,830][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,863][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,896][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:16,897][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:39:16,897][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:39:17,626][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:39:17,923][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:39:18,249][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:39:18,574][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:39:18,897][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:39:19,222][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:39:19,547][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:39:19,871][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:39:20,198][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:39:20,520][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:39:20,845][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:39:21,174][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:39:21,498][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:39:21,830][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:39:22,156][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:39:22,482][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:39:22,811][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:39:23,137][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:39:23,463][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:39:23,789][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:39:24,113][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:39:24,438][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:39:24,762][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:39:25,084][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:39:25,409][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:39:25,733][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:39:26,060][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:39:26,384][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:39:26,709][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:39:27,030][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:39:27,356][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:39:27,679][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:39:28,003][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:39:28,702][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:39:29,450][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:39:29,451][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:39:29,453][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:39:30,329][__main__][INFO] - Iteration 550 took 23s (39.76% Gen, 56.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 48m 47s. Estimated total time: 19h 21m 28s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 42s, 500 more iterations: 3h 13m 34s.
[2025-11-13 11:39:30,331][__main__][INFO] - Starting iteration 550.
[2025-11-13 11:39:30,334][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 54 and human policies 1.
[2025-11-13 11:39:30,334][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:39:39,604][__main__][INFO] - Number of regex retries in iteration 550: 0
[2025-11-13 11:39:39,605][__main__][INFO] - agents played in iteration 550 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:39:40,054][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,088][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,155][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:39:40,156][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:39:40,156][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:39:40,880][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:39:41,176][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:39:41,501][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:39:41,826][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:39:42,152][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:39:42,478][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:39:42,806][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:39:43,130][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:39:43,456][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:39:43,782][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:39:44,114][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:39:44,438][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:39:44,762][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:39:45,090][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:39:45,414][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:39:45,739][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:39:46,067][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:39:46,393][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:39:46,721][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:39:47,049][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:39:47,373][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:39:47,697][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:39:48,023][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:39:48,347][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:39:48,671][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:39:49,000][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:39:49,325][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:39:49,648][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:39:49,971][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:39:50,293][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:39:50,616][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:39:50,939][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:39:51,263][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:39:51,947][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:39:52,670][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:39:52,672][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:39:52,673][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:39:54,436][__main__][INFO] - Iteration 551 took 24s (38.46% Gen, 54.22% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 32m 4s. Estimated total time: 20h 5m 10s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 10s, 500 more iterations: 3h 20m 51s.
[2025-11-13 11:39:54,438][__main__][INFO] - Starting iteration 551.
[2025-11-13 11:39:54,442][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:39:54,442][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:40:03,731][__main__][INFO] - Number of regex retries in iteration 551: 0
[2025-11-13 11:40:03,732][__main__][INFO] - agents played in iteration 551 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:40:04,169][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,203][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:04,272][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:40:04,273][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:40:05,003][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:40:05,300][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:40:05,624][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:40:05,950][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:40:06,277][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:40:06,602][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:40:06,925][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:40:07,252][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:40:07,581][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:40:07,904][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:40:08,231][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:40:08,555][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:40:08,878][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:40:09,203][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:40:09,527][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:40:09,852][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:40:10,177][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:40:10,503][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:40:10,828][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:40:11,153][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:40:11,479][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:40:11,802][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:40:12,127][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:40:12,450][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:40:12,775][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:40:13,100][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:40:13,424][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:40:13,748][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:40:14,073][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:40:14,396][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:40:14,719][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:40:15,042][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:40:15,369][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:40:16,070][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:40:16,800][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:40:16,802][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:40:16,803][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:40:17,659][__main__][INFO] - Iteration 552 took 23s (40.01% Gen, 56.30% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 47m 26s. Estimated total time: 19h 20m 55s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 41s, 500 more iterations: 3h 13m 29s.
[2025-11-13 11:40:17,661][__main__][INFO] - Starting iteration 552.
[2025-11-13 11:40:17,664][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:40:17,665][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:40:27,016][__main__][INFO] - Number of regex retries in iteration 552: 0
[2025-11-13 11:40:27,017][__main__][INFO] - agents played in iteration 552 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:40:27,461][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,529][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,563][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:27,564][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:40:27,564][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:40:28,293][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:40:28,588][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:40:28,913][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:40:29,237][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:40:29,561][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:40:29,886][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:40:30,212][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:40:30,539][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:40:30,867][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:40:31,192][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:40:31,517][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:40:31,840][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:40:32,166][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:40:32,489][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:40:32,812][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:40:33,136][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:40:33,459][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:40:33,783][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:40:34,109][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:40:34,432][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:40:34,756][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:40:35,085][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:40:35,411][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:40:35,737][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:40:36,066][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:40:36,389][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:40:36,712][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:40:37,036][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:40:37,360][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:40:37,688][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:40:38,015][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:40:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:40:38,668][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:40:39,369][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:40:40,087][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:40:40,089][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:40:40,091][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:40:40,956][__main__][INFO] - Iteration 553 took 23s (40.15% Gen, 56.13% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 50m 46s. Estimated total time: 19h 24m 38s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 49s, 500 more iterations: 3h 14m 6s.
[2025-11-13 11:40:40,958][__main__][INFO] - Starting iteration 553.
[2025-11-13 11:40:40,961][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:40:40,962][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:40:50,738][__main__][INFO] - Number of regex retries in iteration 553: 0
[2025-11-13 11:40:50,738][__main__][INFO] - agents played in iteration 553 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:40:51,183][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:51,218][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:51,251][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:51,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:40:51,287][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:40:51,288][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:40:52,005][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:40:52,300][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:40:52,623][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:40:52,949][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:40:53,274][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:40:53,604][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:40:53,931][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:40:54,258][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:40:54,582][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:40:54,908][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:40:55,238][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:40:55,565][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:40:55,889][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:40:56,217][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:40:56,544][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:40:56,867][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:40:57,197][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:40:57,525][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:40:57,855][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:40:58,180][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:40:58,506][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:40:58,832][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:40:59,155][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:40:59,485][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:40:59,814][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:41:00,139][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:41:00,464][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:41:00,789][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:41:01,113][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:41:01,440][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:41:01,767][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:41:02,094][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:41:02,421][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:41:03,133][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:41:03,857][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:41:03,859][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:41:03,860][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:41:04,749][__main__][INFO] - Iteration 554 took 23s (41.09% Gen, 55.16% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 15m 10s. Estimated total time: 19h 49m 26s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 38s, 500 more iterations: 3h 18m 14s.
[2025-11-13 11:41:04,751][__main__][INFO] - Starting iteration 554.
[2025-11-13 11:41:04,754][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:41:04,755][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:41:14,812][__main__][INFO] - Number of regex retries in iteration 554: 0
[2025-11-13 11:41:14,813][__main__][INFO] - agents played in iteration 554 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:41:15,260][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,296][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,330][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,364][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:15,365][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:41:15,365][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:41:16,144][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:41:16,439][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:41:16,766][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:41:17,090][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:41:17,414][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:41:17,741][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:41:18,066][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:41:18,389][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:41:18,715][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:41:19,042][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:41:19,367][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:41:19,691][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:41:20,015][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:41:20,338][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:41:20,663][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:41:20,990][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:41:21,318][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:41:21,642][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:41:21,967][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:41:22,294][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:41:22,616][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:41:22,941][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:41:23,266][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:41:23,590][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:41:23,912][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:41:24,236][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:41:24,562][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:41:24,887][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:41:25,210][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:41:25,535][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:41:25,859][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:41:26,182][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:41:26,508][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:41:27,205][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:41:27,917][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:41:27,919][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:41:27,921][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:41:28,782][__main__][INFO] - Iteration 555 took 24s (41.86% Gen, 54.55% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 26m 46s. Estimated total time: 20h 1m 26s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 2s, 500 more iterations: 3h 20m 14s.
[2025-11-13 11:41:28,784][__main__][INFO] - Starting iteration 555.
[2025-11-13 11:41:28,787][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:41:28,787][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:41:38,239][__main__][INFO] - Number of regex retries in iteration 555: 0
[2025-11-13 11:41:38,240][__main__][INFO] - agents played in iteration 555 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:41:38,679][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,712][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,746][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,780][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:41:38,780][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:41:38,781][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:41:39,491][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:41:39,786][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:41:40,111][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:41:40,433][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:41:40,758][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:41:41,082][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:41:41,406][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:41:41,730][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:41:42,055][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:41:42,381][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:41:42,707][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:41:43,030][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:41:43,355][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:41:43,681][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:41:44,006][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:41:44,331][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:41:44,655][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:41:44,980][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:41:45,304][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:41:45,628][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:41:45,950][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:41:46,275][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:41:46,598][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:41:46,921][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:41:47,246][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:41:47,569][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:41:47,891][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:41:48,214][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:41:48,537][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:41:48,859][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:41:49,184][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:41:49,510][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:41:49,840][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:41:50,556][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:41:51,279][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:41:51,280][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:41:51,282][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:41:52,122][__main__][INFO] - Iteration 556 took 23s (40.50% Gen, 55.89% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 51m 47s. Estimated total time: 19h 26m 50s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 53s, 500 more iterations: 3h 14m 28s.
[2025-11-13 11:41:52,125][__main__][INFO] - Starting iteration 556.
[2025-11-13 11:41:52,127][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:41:52,128][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:42:01,938][__main__][INFO] - Number of regex retries in iteration 556: 0
[2025-11-13 11:42:01,938][__main__][INFO] - agents played in iteration 556 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:42:02,385][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:02,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:02,452][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:02,486][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:02,487][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:42:02,487][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:42:03,182][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:42:03,475][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:42:03,800][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:42:04,125][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:42:04,452][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:42:04,776][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:42:05,102][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:42:05,427][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:42:05,755][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:42:06,078][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:42:06,406][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:42:06,732][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:42:07,056][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:42:07,379][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:42:07,704][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:42:08,034][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:42:08,360][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:42:08,691][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:42:09,015][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:42:09,345][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:42:09,670][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:42:09,994][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:42:10,320][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:42:10,645][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:42:10,970][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:42:11,297][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:42:11,622][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:42:11,946][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:42:12,274][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:42:12,601][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:42:12,931][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:42:13,256][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:42:13,581][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:42:14,281][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:42:15,000][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:42:15,001][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:42:15,003][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:42:15,835][__main__][INFO] - Iteration 557 took 23s (41.37% Gen, 55.11% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 9m 59s. Estimated total time: 19h 45m 26s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 30s, 500 more iterations: 3h 17m 34s.
[2025-11-13 11:42:15,837][__main__][INFO] - Starting iteration 557.
[2025-11-13 11:42:15,840][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:42:15,840][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:42:25,470][__main__][INFO] - Number of regex retries in iteration 557: 0
[2025-11-13 11:42:25,470][__main__][INFO] - agents played in iteration 557 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:42:25,926][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:25,959][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:25,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:26,026][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:26,026][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:42:26,027][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:42:26,732][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:42:27,028][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:42:27,356][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:42:27,683][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:42:28,009][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:42:28,335][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:42:28,665][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:42:28,994][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:42:29,320][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:42:29,647][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:42:29,973][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:42:30,297][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:42:30,622][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:42:30,947][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:42:31,271][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:42:31,596][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:42:31,920][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:42:32,245][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:42:32,568][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:42:32,891][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:42:33,217][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:42:33,542][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:42:33,866][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:42:34,189][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:42:34,514][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:42:34,838][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:42:35,163][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:42:35,485][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:42:35,809][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:42:36,134][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:42:36,460][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:42:36,786][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:42:37,110][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:42:37,805][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:42:38,528][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:42:38,530][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:42:38,531][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:42:39,330][__main__][INFO] - Iteration 558 took 23s (40.99% Gen, 55.60% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 58m 43s. Estimated total time: 19h 34m 33s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 9s, 500 more iterations: 3h 15m 45s.
[2025-11-13 11:42:39,333][__main__][INFO] - Starting iteration 558.
[2025-11-13 11:42:39,335][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:42:39,336][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:42:48,844][__main__][INFO] - Number of regex retries in iteration 558: 0
[2025-11-13 11:42:48,845][__main__][INFO] - agents played in iteration 558 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:42:49,302][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:49,336][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:49,370][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:49,403][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:42:49,404][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:42:49,405][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:42:50,131][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:42:50,426][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:42:50,750][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:42:51,074][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:42:51,402][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:42:51,727][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:42:52,051][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:42:52,376][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:42:52,705][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:42:53,031][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:42:53,353][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:42:53,677][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:42:54,001][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:42:54,328][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:42:54,652][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:42:54,976][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:42:55,300][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:42:55,626][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:42:55,949][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:42:56,272][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:42:56,597][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:42:56,924][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:42:57,253][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:42:57,579][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:42:57,909][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:42:58,241][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:42:58,570][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:42:58,900][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:42:59,231][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:42:59,560][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:42:59,889][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:43:00,220][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:43:00,546][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:43:01,234][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:43:01,944][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:43:01,946][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:43:01,947][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:43:02,904][__main__][INFO] - Iteration 559 took 23s (40.34% Gen, 55.59% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 2m 14s. Estimated total time: 19h 38m 28s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 16s, 500 more iterations: 3h 16m 24s.
[2025-11-13 11:43:02,906][__main__][INFO] - Starting iteration 559.
[2025-11-13 11:43:02,908][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:43:02,909][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:43:12,524][__main__][INFO] - Number of regex retries in iteration 559: 0
[2025-11-13 11:43:12,525][__main__][INFO] - agents played in iteration 559 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:43:12,970][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:13,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:13,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:13,071][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:13,072][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:43:13,072][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:43:13,787][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:43:14,084][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:43:14,411][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:43:14,740][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:43:15,067][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:43:15,394][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:43:15,720][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:43:16,049][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:43:16,378][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:43:16,701][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:43:17,025][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:43:17,350][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:43:17,676][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:43:18,000][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:43:18,328][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:43:18,654][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:43:18,980][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:43:19,305][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:43:19,629][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:43:19,952][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:43:20,282][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:43:20,612][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:43:20,936][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:43:21,259][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:43:21,584][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:43:21,913][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:43:22,237][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:43:22,559][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:43:22,887][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:43:23,217][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:43:23,548][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:43:23,877][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:43:24,204][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:43:24,902][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:43:25,616][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:43:25,617][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:43:25,619][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:43:26,539][__main__][INFO] - Iteration 560 took 23s (40.69% Gen, 55.41% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 4m 55s. Estimated total time: 19h 41m 33s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 23s, 500 more iterations: 3h 16m 55s.
[2025-11-13 11:43:26,541][__main__][INFO] - Starting iteration 560.
[2025-11-13 11:43:26,544][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 55 and human policies 1.
[2025-11-13 11:43:26,545][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:43:36,519][__main__][INFO] - Number of regex retries in iteration 560: 0
[2025-11-13 11:43:36,520][__main__][INFO] - agents played in iteration 560 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:43:36,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:36,999][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:37,032][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:37,067][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:43:37,067][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:43:37,068][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:43:37,778][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:43:38,074][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:43:38,405][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:43:38,730][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:43:39,054][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:43:39,380][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:43:39,706][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:43:40,031][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:43:40,354][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:43:40,683][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:43:41,010][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:43:41,333][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:43:41,660][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:43:41,987][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:43:42,311][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:43:42,635][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:43:42,960][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:43:43,283][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:43:43,608][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:43:43,930][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:43:44,254][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:43:44,583][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:43:44,906][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:43:45,234][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:43:45,564][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:43:45,890][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:43:46,214][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:43:46,538][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:43:46,861][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:43:47,185][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:43:47,511][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:43:47,835][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:43:48,159][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:43:48,850][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:43:49,575][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:43:49,577][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:43:49,578][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:43:51,275][__main__][INFO] - Iteration 561 took 24s (40.33% Gen, 52.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 59m 32s. Estimated total time: 20h 36m 34s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 13s, 500 more iterations: 3h 26m 5s.
[2025-11-13 11:43:51,277][__main__][INFO] - Starting iteration 561.
[2025-11-13 11:43:51,280][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:43:51,280][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:44:01,097][__main__][INFO] - Number of regex retries in iteration 561: 0
[2025-11-13 11:44:01,097][__main__][INFO] - agents played in iteration 561 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:44:01,538][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:01,571][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:01,603][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:01,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:01,637][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:44:01,637][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:44:02,325][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:44:02,620][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:44:02,946][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:44:03,270][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:44:03,597][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:44:03,922][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:44:04,251][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:44:04,578][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:44:04,903][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:44:05,230][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:44:05,553][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:44:05,880][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:44:06,207][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:44:06,531][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:44:06,856][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:44:07,184][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:44:07,509][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:44:07,831][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:44:08,154][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:44:08,477][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:44:08,801][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:44:09,128][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:44:09,457][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:44:09,786][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:44:10,110][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:44:10,434][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:44:10,759][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:44:11,084][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:44:11,409][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:44:11,733][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:44:12,057][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:44:12,381][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:44:12,706][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:44:13,401][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:44:14,108][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:44:14,109][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:44:14,111][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:44:14,959][__main__][INFO] - Iteration 562 took 23s (41.45% Gen, 54.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 6m 36s. Estimated total time: 19h 44m 2s. Time estimates for 10 more iterations: 3m 56s, 100 more iterations: 39m 28s, 500 more iterations: 3h 17m 20s.
[2025-11-13 11:44:14,961][__main__][INFO] - Starting iteration 562.
[2025-11-13 11:44:14,964][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:44:14,964][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:44:24,245][__main__][INFO] - Number of regex retries in iteration 562: 0
[2025-11-13 11:44:24,246][__main__][INFO] - agents played in iteration 562 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:44:24,689][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:25,044][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:25,077][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:25,110][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:25,110][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:44:25,111][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:44:25,812][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:44:26,105][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:44:26,430][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:44:26,754][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:44:27,077][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:44:27,401][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:44:27,727][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:44:28,052][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:44:28,381][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:44:28,708][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:44:29,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:44:29,359][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:44:29,688][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:44:30,015][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:44:30,342][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:44:30,672][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:44:30,997][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:44:31,322][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:44:31,647][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:44:31,972][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:44:32,302][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:44:32,626][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:44:32,950][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:44:33,278][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:44:33,602][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:44:33,924][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:44:34,253][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:44:34,581][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:44:34,911][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:44:35,236][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:44:35,566][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:44:35,897][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:44:36,226][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:44:36,925][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:44:37,610][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:44:37,611][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:44:37,612][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:44:38,408][__main__][INFO] - Iteration 563 took 23s (39.59% Gen, 57.01% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 54m 24s. Estimated total time: 19h 32m 13s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 4s, 500 more iterations: 3h 15m 22s.
[2025-11-13 11:44:38,410][__main__][INFO] - Starting iteration 563.
[2025-11-13 11:44:38,413][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:44:38,413][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:44:48,335][__main__][INFO] - Number of regex retries in iteration 563: 0
[2025-11-13 11:44:48,336][__main__][INFO] - agents played in iteration 563 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:44:48,789][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:48,822][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:48,854][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:48,887][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:44:48,888][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:44:48,888][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:44:49,576][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:44:49,870][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:44:50,199][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:44:50,529][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:44:50,855][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:44:51,179][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:44:51,510][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:44:51,839][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:44:52,169][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:44:52,497][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:44:52,825][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:44:53,148][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:44:53,476][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:44:53,803][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:44:54,131][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:44:54,460][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:44:54,786][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:44:55,115][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:44:55,440][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:44:55,765][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:44:56,091][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:44:56,414][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:44:56,738][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:44:57,067][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:44:57,393][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:44:57,724][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:44:58,055][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:44:58,379][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:44:58,705][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:44:59,032][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:44:59,359][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:44:59,688][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:45:00,017][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:45:00,724][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:45:01,437][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:45:01,439][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:45:01,441][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:45:02,290][__main__][INFO] - Iteration 564 took 23s (41.55% Gen, 54.88% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 15m 42s. Estimated total time: 19h 53m 55s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 47s, 500 more iterations: 3h 18m 59s.
[2025-11-13 11:45:02,292][__main__][INFO] - Starting iteration 564.
[2025-11-13 11:45:02,295][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:45:02,296][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:45:12,010][__main__][INFO] - Number of regex retries in iteration 564: 0
[2025-11-13 11:45:12,011][__main__][INFO] - agents played in iteration 564 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:45:12,456][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:12,490][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:12,523][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:12,556][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:12,557][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:45:12,558][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:45:13,250][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:45:13,542][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:45:13,867][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:45:14,192][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:45:14,519][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:45:14,846][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:45:15,169][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:45:15,493][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:45:15,818][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:45:16,148][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:45:16,472][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:45:16,797][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:45:17,123][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:45:17,448][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:45:17,775][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:45:18,103][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:45:18,428][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:45:18,757][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:45:19,082][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:45:19,410][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:45:19,732][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:45:20,057][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:45:20,384][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:45:20,712][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:45:21,039][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:45:21,363][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:45:21,689][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:45:22,011][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:45:22,341][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:45:22,670][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:45:22,998][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:45:23,321][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:45:23,646][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:45:24,342][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:45:25,060][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:45:25,062][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:45:25,063][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:45:25,881][__main__][INFO] - Iteration 565 took 23s (41.19% Gen, 55.34% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 0m 42s. Estimated total time: 19h 39m 19s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 18s, 500 more iterations: 3h 16m 33s.
[2025-11-13 11:45:25,883][__main__][INFO] - Starting iteration 565.
[2025-11-13 11:45:25,885][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:45:25,886][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:45:35,497][__main__][INFO] - Number of regex retries in iteration 565: 0
[2025-11-13 11:45:35,497][__main__][INFO] - agents played in iteration 565 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:45:35,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:35,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:36,008][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:36,041][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:36,042][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:45:36,044][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:45:36,809][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:45:37,104][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:45:37,429][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:45:37,754][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:45:38,079][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:45:38,404][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:45:38,731][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:45:39,055][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:45:39,379][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:45:39,703][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:45:40,027][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:45:40,351][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:45:40,674][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:45:40,998][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:45:41,324][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:45:41,653][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:45:41,980][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:45:42,305][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:45:42,628][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:45:42,951][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:45:43,273][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:45:43,596][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:45:43,920][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:45:44,245][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:45:44,569][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:45:44,893][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:45:45,217][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:45:45,542][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:45:45,867][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:45:46,191][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:45:46,516][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:45:46,847][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:45:47,172][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:45:47,867][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:45:48,591][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:45:48,592][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:45:48,594][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:45:49,410][__main__][INFO] - Iteration 566 took 23s (40.86% Gen, 55.66% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 57m 17s. Estimated total time: 19h 36m 18s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 3s.
[2025-11-13 11:45:49,412][__main__][INFO] - Starting iteration 566.
[2025-11-13 11:45:49,415][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:45:49,415][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:45:58,665][__main__][INFO] - Number of regex retries in iteration 566: 0
[2025-11-13 11:45:58,666][__main__][INFO] - agents played in iteration 566 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:45:59,114][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:59,147][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:59,180][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:59,213][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:45:59,214][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:45:59,214][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:45:59,931][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:46:00,227][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:46:00,551][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:46:00,874][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:46:01,198][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:46:01,529][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:46:01,856][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:46:02,186][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:46:02,513][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:46:02,841][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:46:03,166][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:46:03,492][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:46:03,819][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:46:04,142][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:46:04,467][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:46:04,791][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:46:05,117][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:46:05,443][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:46:05,771][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:46:06,100][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:46:06,426][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:46:06,753][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:46:07,079][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:46:07,404][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:46:07,726][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:46:08,051][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:46:08,375][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:46:08,704][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:46:09,031][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:46:09,358][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:46:09,685][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:46:10,009][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:46:10,333][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:46:11,026][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:46:11,757][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:46:11,759][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:46:11,761][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:46:12,644][__main__][INFO] - Iteration 567 took 23s (39.82% Gen, 56.37% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 42m 6s. Estimated total time: 19h 21m 29s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 42s, 500 more iterations: 3h 13m 34s.
[2025-11-13 11:46:12,647][__main__][INFO] - Starting iteration 567.
[2025-11-13 11:46:12,650][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:46:12,650][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:46:21,507][__main__][INFO] - Number of regex retries in iteration 567: 0
[2025-11-13 11:46:21,508][__main__][INFO] - agents played in iteration 567 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:46:21,981][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,049][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,082][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:22,083][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:46:22,084][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:46:22,811][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:46:23,107][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:46:23,432][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:46:23,757][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:46:24,084][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:46:24,409][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:46:24,739][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:46:25,068][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:46:25,398][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:46:25,725][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:46:26,055][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:46:26,381][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:46:26,711][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:46:27,035][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:46:27,361][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:46:27,684][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:46:28,010][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:46:28,335][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:46:28,660][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:46:28,990][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:46:29,317][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:46:29,638][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:46:29,964][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:46:30,287][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:46:30,617][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:46:30,942][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:46:31,267][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:46:31,590][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:46:31,914][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:46:32,240][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:46:32,562][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:46:32,891][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:46:33,216][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:46:33,923][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:46:34,646][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:46:34,647][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:46:34,649][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:46:35,679][__main__][INFO] - Iteration 568 took 23s (38.46% Gen, 57.06% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 31m 43s. Estimated total time: 19h 11m 30s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 55s.
[2025-11-13 11:46:35,681][__main__][INFO] - Starting iteration 568.
[2025-11-13 11:46:35,684][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:46:35,685][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:46:45,917][__main__][INFO] - Number of regex retries in iteration 568: 0
[2025-11-13 11:46:45,918][__main__][INFO] - agents played in iteration 568 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:46:46,359][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:46,393][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:46,426][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:46,460][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:46:46,460][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:46:46,461][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:46:47,181][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:46:47,476][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:46:47,805][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:46:48,130][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:46:48,454][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:46:48,780][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:46:49,104][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:46:49,430][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:46:49,757][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:46:50,085][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:46:50,410][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:46:50,737][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:46:51,063][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:46:51,390][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:46:51,715][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:46:52,040][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:46:52,367][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:46:52,694][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:46:53,015][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:46:53,342][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:46:53,665][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:46:53,987][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:46:54,311][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:46:54,635][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:46:54,957][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:46:55,282][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:46:55,606][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:46:55,929][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:46:56,253][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:46:56,579][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:46:56,903][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:46:57,231][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:46:57,556][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:46:58,248][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:46:58,968][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:46:58,969][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:46:58,971][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:46:59,798][__main__][INFO] - Iteration 569 took 24s (42.43% Gen, 54.13% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 25m 34s. Estimated total time: 20h 5m 45s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 11s, 500 more iterations: 3h 20m 57s.
[2025-11-13 11:46:59,800][__main__][INFO] - Starting iteration 569.
[2025-11-13 11:46:59,804][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:46:59,804][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:47:09,188][__main__][INFO] - Number of regex retries in iteration 569: 0
[2025-11-13 11:47:09,188][__main__][INFO] - agents played in iteration 569 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:47:09,637][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:09,670][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:09,703][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:09,736][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:09,737][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:47:09,737][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:47:10,451][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:47:10,746][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:47:11,071][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:47:11,395][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:47:11,718][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:47:12,043][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:47:12,367][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:47:12,692][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:47:13,017][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:47:13,340][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:47:13,666][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:47:13,994][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:47:14,321][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:47:14,646][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:47:14,972][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:47:15,297][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:47:15,620][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:47:15,945][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:47:16,274][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:47:16,601][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:47:16,930][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:47:17,256][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:47:17,581][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:47:17,905][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:47:18,230][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:47:18,554][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:47:18,877][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:47:19,207][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:47:19,532][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:47:19,856][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:47:20,185][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:47:20,510][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:47:20,835][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:47:21,525][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:47:22,242][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:47:22,243][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:47:22,245][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:47:23,090][__main__][INFO] - Iteration 570 took 23s (40.30% Gen, 56.07% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 43m 48s. Estimated total time: 19h 24m 22s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 3s.
[2025-11-13 11:47:23,092][__main__][INFO] - Starting iteration 570.
[2025-11-13 11:47:23,095][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 56 and human policies 1.
[2025-11-13 11:47:23,095][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:47:32,470][__main__][INFO] - Number of regex retries in iteration 570: 0
[2025-11-13 11:47:32,471][__main__][INFO] - agents played in iteration 570 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:47:32,912][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:32,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:32,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:33,013][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:33,014][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:47:33,014][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:47:33,729][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:47:34,024][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:47:34,350][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:47:34,674][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:47:35,000][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:47:35,324][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:47:35,649][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:47:35,974][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:47:36,299][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:47:36,623][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:47:36,947][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:47:37,271][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:47:37,596][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:47:37,921][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:47:38,246][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:47:38,569][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:47:38,893][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:47:39,219][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:47:39,541][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:47:39,866][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:47:40,188][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:47:40,512][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:47:40,837][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:47:41,162][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:47:41,489][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:47:41,812][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:47:42,135][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:47:42,459][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:47:42,783][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:47:43,108][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:47:43,432][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:47:43,757][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:47:44,081][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:47:44,765][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:47:45,486][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:47:45,487][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:47:45,489][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:47:47,215][__main__][INFO] - Iteration 571 took 24s (38.87% Gen, 53.97% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 25m 4s. Estimated total time: 20h 6m 3s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 12s, 500 more iterations: 3h 21m 0s.
[2025-11-13 11:47:47,216][__main__][INFO] - Starting iteration 571.
[2025-11-13 11:47:47,219][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:47:47,220][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:47:56,407][__main__][INFO] - Number of regex retries in iteration 571: 0
[2025-11-13 11:47:56,408][__main__][INFO] - agents played in iteration 571 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:47:56,848][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:56,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:56,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:56,947][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:47:56,948][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:47:56,948][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:47:57,666][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:47:57,961][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:47:58,286][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:47:58,616][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:47:58,940][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:47:59,265][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:47:59,591][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:47:59,917][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:48:00,242][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:48:00,567][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:48:00,892][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:48:01,218][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:48:01,544][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:48:01,868][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:48:02,194][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:48:02,519][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:48:02,844][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:48:03,169][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:48:03,493][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:48:03,817][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:48:04,143][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:48:04,468][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:48:04,791][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:48:05,114][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:48:05,439][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:48:05,766][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:48:06,092][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:48:06,422][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:48:06,748][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:48:07,075][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:48:07,402][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:48:07,730][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:48:08,054][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:48:08,770][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:48:09,492][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:48:09,494][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:48:09,495][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:48:10,374][__main__][INFO] - Iteration 572 took 23s (39.68% Gen, 56.52% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 36m 25s. Estimated total time: 19h 17m 46s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 35s, 500 more iterations: 3h 12m 57s.
[2025-11-13 11:48:10,376][__main__][INFO] - Starting iteration 572.
[2025-11-13 11:48:10,379][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:48:10,379][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:48:19,492][__main__][INFO] - Number of regex retries in iteration 572: 0
[2025-11-13 11:48:19,493][__main__][INFO] - agents played in iteration 572 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:48:19,936][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:19,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:20,001][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:20,034][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:20,035][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:48:20,036][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:48:20,753][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:48:21,049][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:48:21,378][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:48:21,707][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:48:22,031][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:48:22,355][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:48:22,680][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:48:23,007][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:48:23,330][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:48:23,657][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:48:23,982][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:48:24,306][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:48:24,631][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:48:24,954][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:48:25,280][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:48:25,605][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:48:25,929][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:48:26,254][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:48:26,576][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:48:26,899][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:48:27,223][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:48:27,547][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:48:27,871][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:48:28,193][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:48:28,517][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:48:28,840][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:48:29,165][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:48:29,489][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:48:29,813][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:48:30,137][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:48:30,462][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:48:30,786][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:48:31,112][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:48:31,815][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:48:32,529][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:48:32,531][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:48:32,532][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:48:33,370][__main__][INFO] - Iteration 573 took 22s (39.63% Gen, 56.71% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 27m 51s. Estimated total time: 19h 9m 36s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 19s, 500 more iterations: 3h 11m 36s.
[2025-11-13 11:48:33,372][__main__][INFO] - Starting iteration 573.
[2025-11-13 11:48:33,374][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:48:33,375][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:48:43,612][__main__][INFO] - Number of regex retries in iteration 573: 0
[2025-11-13 11:48:43,612][__main__][INFO] - agents played in iteration 573 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:48:44,050][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:44,083][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:44,116][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:44,149][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:48:44,150][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:48:44,150][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:48:44,832][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:48:45,127][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:48:45,452][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:48:45,780][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:48:46,107][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:48:46,436][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:48:46,760][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:48:47,085][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:48:47,410][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:48:47,738][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:48:48,064][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:48:48,390][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:48:48,716][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:48:49,039][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:48:49,363][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:48:49,689][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:48:50,013][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:48:50,338][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:48:50,660][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:48:50,985][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:48:51,308][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:48:51,632][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:48:51,956][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:48:52,279][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:48:52,604][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:48:52,928][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:48:53,252][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:48:53,580][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:48:53,904][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:48:54,228][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:48:54,554][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:48:54,879][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:48:55,204][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:48:55,879][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:48:56,598][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:48:56,599][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:48:56,601][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:48:57,461][__main__][INFO] - Iteration 574 took 24s (42.50% Gen, 53.92% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 22m 14s. Estimated total time: 20h 4m 22s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 8s, 500 more iterations: 3h 20m 43s.
[2025-11-13 11:48:57,463][__main__][INFO] - Starting iteration 574.
[2025-11-13 11:48:57,466][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:48:57,466][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:49:08,002][__main__][INFO] - Number of regex retries in iteration 574: 0
[2025-11-13 11:49:08,002][__main__][INFO] - agents played in iteration 574 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:49:08,449][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,482][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,515][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,548][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:08,549][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:49:08,550][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:49:09,273][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:49:09,569][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:49:09,894][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:49:10,219][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:49:10,543][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:49:10,868][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:49:11,195][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:49:11,522][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:49:11,846][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:49:12,175][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:49:12,502][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:49:12,828][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:49:13,153][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:49:13,479][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:49:13,808][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:49:14,134][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:49:14,462][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:49:14,788][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:49:15,112][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:49:15,437][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:49:15,764][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:49:16,087][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:49:16,413][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:49:16,738][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:49:17,068][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:49:17,394][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:49:17,720][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:49:18,049][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:49:18,375][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:49:18,705][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:49:19,032][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:49:19,358][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:49:19,685][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:49:20,378][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:49:21,088][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:49:21,090][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:49:21,091][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:49:21,990][__main__][INFO] - Iteration 575 took 24s (42.96% Gen, 53.37% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 43m 41s. Estimated total time: 20h 26m 14s. Time estimates for 10 more iterations: 4m 5s, 100 more iterations: 40m 52s, 500 more iterations: 3h 24m 22s.
[2025-11-13 11:49:21,992][__main__][INFO] - Starting iteration 575.
[2025-11-13 11:49:21,995][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:49:21,996][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:49:31,377][__main__][INFO] - Number of regex retries in iteration 575: 0
[2025-11-13 11:49:31,377][__main__][INFO] - agents played in iteration 575 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:49:31,818][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:31,851][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:31,885][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:31,919][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:31,919][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:49:31,920][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:49:32,640][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:49:32,937][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:49:33,262][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:49:33,587][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:49:33,913][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:49:34,237][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:49:34,560][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:49:34,884][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:49:35,208][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:49:35,533][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:49:35,863][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:49:36,188][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:49:36,516][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:49:36,839][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:49:37,163][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:49:37,487][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:49:37,811][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:49:38,137][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:49:38,460][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:49:38,784][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:49:39,110][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:49:39,433][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:49:39,757][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:49:40,080][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:49:40,404][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:49:40,729][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:49:41,054][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:49:41,376][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:49:41,699][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:49:42,023][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:49:42,345][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:49:42,668][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:49:42,992][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:49:43,679][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:49:44,396][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:49:44,397][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:49:44,399][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:49:45,281][__main__][INFO] - Iteration 576 took 23s (40.28% Gen, 55.92% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 41m 23s. Estimated total time: 19h 24m 20s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 48s, 500 more iterations: 3h 14m 3s.
[2025-11-13 11:49:45,283][__main__][INFO] - Starting iteration 576.
[2025-11-13 11:49:45,287][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:49:45,287][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:49:54,419][__main__][INFO] - Number of regex retries in iteration 576: 0
[2025-11-13 11:49:54,420][__main__][INFO] - agents played in iteration 576 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:49:54,862][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:54,895][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:54,928][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:54,961][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:49:54,962][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:49:54,963][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:49:55,679][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:49:55,974][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:49:56,301][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:49:56,629][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:49:56,955][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:49:57,286][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:49:57,611][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:49:57,936][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:49:58,265][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:49:58,594][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:49:58,919][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:49:59,244][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:49:59,570][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:49:59,894][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:50:00,225][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:50:00,548][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:50:00,873][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:50:01,199][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:50:01,524][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:50:01,850][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:50:02,179][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:50:02,506][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:50:02,833][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:50:03,162][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:50:03,488][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:50:03,814][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:50:04,144][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:50:04,474][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:50:04,802][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:50:05,129][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:50:05,457][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:50:05,788][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:50:06,115][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:50:06,785][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:50:07,500][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:50:07,501][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:50:07,503][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:50:08,368][__main__][INFO] - Iteration 577 took 23s (39.56% Gen, 56.68% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 30m 47s. Estimated total time: 19h 14m 6s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 28s, 500 more iterations: 3h 12m 21s.
[2025-11-13 11:50:08,370][__main__][INFO] - Starting iteration 577.
[2025-11-13 11:50:08,374][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:50:08,374][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:50:18,258][__main__][INFO] - Number of regex retries in iteration 577: 0
[2025-11-13 11:50:18,259][__main__][INFO] - agents played in iteration 577 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:50:18,706][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:18,739][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:18,772][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:18,806][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:18,806][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:50:18,806][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:50:19,516][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:50:19,810][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:50:20,134][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:50:20,458][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:50:20,784][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:50:21,109][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:50:21,434][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:50:21,758][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:50:22,082][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:50:22,407][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:50:22,731][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:50:23,056][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:50:23,380][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:50:23,706][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:50:24,031][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:50:24,357][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:50:24,681][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:50:25,006][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:50:25,329][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:50:25,655][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:50:25,983][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:50:26,312][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:50:26,637][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:50:26,966][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:50:27,297][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:50:27,626][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:50:27,955][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:50:28,281][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:50:28,607][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:50:28,933][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:50:29,259][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:50:29,588][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:50:29,915][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:50:30,587][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:50:31,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:50:31,318][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:50:31,320][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:50:32,192][__main__][INFO] - Iteration 578 took 23s (41.50% Gen, 54.83% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 7m 15s. Estimated total time: 19h 50m 58s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 41s, 500 more iterations: 3h 18m 29s.
[2025-11-13 11:50:32,194][__main__][INFO] - Starting iteration 578.
[2025-11-13 11:50:32,198][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:50:32,199][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:50:42,117][__main__][INFO] - Number of regex retries in iteration 578: 0
[2025-11-13 11:50:42,118][__main__][INFO] - agents played in iteration 578 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:50:42,581][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:42,614][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:42,647][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:42,680][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:50:42,681][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:50:42,682][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:50:43,390][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:50:43,688][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:50:44,016][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:50:44,348][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:50:44,680][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:50:45,003][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:50:45,328][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:50:45,656][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:50:45,981][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:50:46,306][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:50:46,630][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:50:46,954][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:50:47,278][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:50:47,602][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:50:47,932][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:50:48,261][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:50:48,588][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:50:48,916][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:50:49,241][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:50:49,566][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:50:49,892][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:50:50,217][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:50:50,543][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:50:50,873][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:50:51,200][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:50:51,524][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:50:51,847][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:50:52,169][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:50:52,492][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:50:52,818][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:50:53,146][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:50:53,475][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:50:53,799][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:50:54,471][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:50:55,202][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:50:55,204][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:50:55,206][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:50:56,068][__main__][INFO] - Iteration 579 took 23s (41.55% Gen, 54.83% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 9m 27s. Estimated total time: 19h 53m 35s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 47s, 500 more iterations: 3h 18m 55s.
[2025-11-13 11:50:56,070][__main__][INFO] - Starting iteration 579.
[2025-11-13 11:50:56,074][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:50:56,074][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:51:06,032][__main__][INFO] - Number of regex retries in iteration 579: 0
[2025-11-13 11:51:06,032][__main__][INFO] - agents played in iteration 579 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:51:06,473][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:06,507][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:06,540][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:06,574][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:06,575][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:51:06,575][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:51:07,317][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:51:07,613][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:51:07,938][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:51:08,266][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:51:08,590][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:51:08,914][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:51:09,242][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:51:09,568][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:51:09,895][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:51:10,219][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:51:10,544][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:51:10,872][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:51:11,197][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:51:11,521][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:51:11,845][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:51:12,170][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:51:12,493][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:51:12,818][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:51:13,142][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:51:13,467][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:51:13,790][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:51:14,114][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:51:14,438][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:51:14,764][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:51:15,088][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:51:15,411][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:51:15,739][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:51:16,065][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:51:16,393][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:51:16,719][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:51:17,046][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:51:17,372][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:51:17,700][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:51:18,446][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:51:19,178][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:51:19,180][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:51:19,181][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:51:20,091][__main__][INFO] - Iteration 580 took 24s (41.46% Gen, 54.74% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 16m 23s. Estimated total time: 20h 0m 54s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 1s, 500 more iterations: 3h 20m 9s.
[2025-11-13 11:51:20,093][__main__][INFO] - Starting iteration 580.
[2025-11-13 11:51:20,096][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 57 and human policies 1.
[2025-11-13 11:51:20,097][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:51:30,430][__main__][INFO] - Number of regex retries in iteration 580: 0
[2025-11-13 11:51:30,430][__main__][INFO] - agents played in iteration 580 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:51:30,876][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:30,909][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:30,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:30,975][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:30,975][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:51:30,975][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:51:31,715][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:51:32,011][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:51:32,336][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:51:32,659][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:51:32,984][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:51:33,307][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:51:33,630][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:51:33,953][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:51:34,277][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:51:34,602][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:51:34,928][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:51:35,252][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:51:35,577][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:51:35,903][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:51:36,234][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:51:36,563][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:51:36,888][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:51:37,212][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:51:37,534][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:51:37,857][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:51:38,181][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:51:38,506][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:51:38,831][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:51:39,154][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:51:39,479][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:51:39,808][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:51:40,132][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:51:40,459][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:51:40,785][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:51:41,112][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:51:41,440][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:51:41,767][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:51:42,095][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:51:42,778][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:51:43,519][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:51:43,521][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:51:43,523][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:51:45,452][__main__][INFO] - Iteration 581 took 25s (40.75% Gen, 51.63% Train). Generation: 10s, Training: 13s. Estimated remaining time: 20h 22m 54s. Estimated total time: 21h 7m 50s. Time estimates for 10 more iterations: 4m 13s, 100 more iterations: 42m 15s, 500 more iterations: 3h 31m 18s.
[2025-11-13 11:51:45,455][__main__][INFO] - Starting iteration 581.
[2025-11-13 11:51:45,458][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:51:45,459][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:51:55,649][__main__][INFO] - Number of regex retries in iteration 581: 0
[2025-11-13 11:51:55,650][__main__][INFO] - agents played in iteration 581 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:51:56,086][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,122][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,156][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,189][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:51:56,190][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:51:56,190][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:51:56,910][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:51:57,206][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:51:57,530][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:51:57,854][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:51:58,177][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:51:58,507][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:51:58,835][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:51:59,160][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:51:59,487][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:51:59,818][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:52:00,142][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:52:00,464][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:52:00,789][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:52:01,117][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:52:01,439][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:52:01,766][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:52:02,090][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:52:02,415][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:52:02,741][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:52:03,064][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:52:03,388][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:52:03,712][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:52:04,035][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:52:04,357][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:52:04,681][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:52:05,006][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:52:05,329][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:52:05,654][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:52:05,978][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:52:06,305][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:52:06,631][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:52:06,957][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:52:07,280][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:52:07,985][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:52:08,736][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:52:08,737][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:52:08,739][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:52:09,628][__main__][INFO] - Iteration 582 took 24s (42.16% Gen, 54.15% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 23m 12s. Estimated total time: 20h 8m 32s. Time estimates for 10 more iterations: 4m 1s, 100 more iterations: 40m 17s, 500 more iterations: 3h 21m 25s.
[2025-11-13 11:52:09,631][__main__][INFO] - Starting iteration 582.
[2025-11-13 11:52:09,634][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:52:09,635][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:52:19,469][__main__][INFO] - Number of regex retries in iteration 582: 0
[2025-11-13 11:52:19,469][__main__][INFO] - agents played in iteration 582 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:52:19,911][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:19,947][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:19,980][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:20,015][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:20,015][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:52:20,016][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:52:20,747][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:52:21,043][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:52:21,368][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:52:21,692][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:52:22,015][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:52:22,342][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:52:22,667][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:52:22,999][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:52:23,323][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:52:23,651][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:52:23,980][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:52:24,311][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:52:24,634][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:52:24,960][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:52:25,285][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:52:25,612][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:52:25,937][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:52:26,267][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:52:26,593][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:52:26,919][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:52:27,246][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:52:27,570][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:52:27,894][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:52:28,217][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:52:28,540][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:52:28,865][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:52:29,190][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:52:29,514][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:52:29,837][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:52:30,159][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:52:30,483][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:52:30,808][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:52:31,132][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:52:31,818][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:52:32,557][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:52:32,559][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:52:32,561][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:52:33,658][__main__][INFO] - Iteration 583 took 24s (40.93% Gen, 54.49% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 15m 31s. Estimated total time: 20h 1m 15s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 2s, 500 more iterations: 3h 20m 12s.
[2025-11-13 11:52:33,661][__main__][INFO] - Starting iteration 583.
[2025-11-13 11:52:33,663][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:52:33,664][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:52:44,433][__main__][INFO] - Number of regex retries in iteration 583: 0
[2025-11-13 11:52:44,433][__main__][INFO] - agents played in iteration 583 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:52:44,872][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:44,907][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:44,941][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:44,974][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:52:44,974][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:52:44,975][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:52:45,715][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:52:46,012][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:52:46,336][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:52:46,660][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:52:46,987][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:52:47,314][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:52:47,638][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:52:47,963][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:52:48,292][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:52:48,618][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:52:48,948][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:52:49,277][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:52:49,607][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:52:49,933][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:52:50,259][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:52:50,587][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:52:50,913][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:52:51,243][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:52:51,570][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:52:51,898][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:52:52,223][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:52:52,548][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:52:52,871][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:52:53,195][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:52:53,521][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:52:53,851][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:52:54,178][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:52:54,502][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:52:54,827][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:52:55,157][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:52:55,485][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:52:55,811][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:52:56,137][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:52:56,856][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:52:57,586][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:52:57,587][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:52:57,589][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:52:58,408][__main__][INFO] - Iteration 584 took 24s (43.52% Gen, 53.17% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 51m 6s. Estimated total time: 20h 37m 16s. Time estimates for 10 more iterations: 4m 7s, 100 more iterations: 41m 14s, 500 more iterations: 3h 26m 12s.
[2025-11-13 11:52:58,410][__main__][INFO] - Starting iteration 584.
[2025-11-13 11:52:58,412][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:52:58,413][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:53:08,418][__main__][INFO] - Number of regex retries in iteration 584: 0
[2025-11-13 11:53:08,418][__main__][INFO] - agents played in iteration 584 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:53:08,847][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:08,880][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:08,913][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:08,946][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:08,947][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:53:08,947][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:53:09,636][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:53:09,932][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:53:10,259][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:53:10,589][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:53:10,912][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:53:11,238][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:53:11,565][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:53:11,888][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:53:12,214][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:53:12,542][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:53:12,866][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:53:13,192][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:53:13,515][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:53:13,841][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:53:14,164][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:53:14,489][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:53:14,814][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:53:15,137][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:53:15,462][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:53:15,787][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:53:16,112][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:53:16,436][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:53:16,759][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:53:17,083][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:53:17,408][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:53:17,732][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:53:18,056][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:53:18,378][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:53:18,703][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:53:19,028][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:53:19,353][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:53:19,678][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:53:20,003][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:53:20,692][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:53:21,408][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:53:21,409][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:53:21,411][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:53:22,207][__main__][INFO] - Iteration 585 took 23s (42.04% Gen, 54.60% Train). Generation: 10s, Training: 12s. Estimated remaining time: 19h 3m 13s. Estimated total time: 19h 49m 46s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 39s, 500 more iterations: 3h 18m 17s.
[2025-11-13 11:53:22,209][__main__][INFO] - Starting iteration 585.
[2025-11-13 11:53:22,212][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:53:22,213][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:53:31,638][__main__][INFO] - Number of regex retries in iteration 585: 0
[2025-11-13 11:53:31,639][__main__][INFO] - agents played in iteration 585 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:53:32,072][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:32,108][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:32,141][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:32,174][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:32,175][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:53:32,175][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:53:32,870][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:53:33,165][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:53:33,490][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:53:33,814][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:53:34,142][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:53:34,465][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:53:34,792][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:53:35,118][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:53:35,441][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:53:35,767][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:53:36,092][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:53:36,417][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:53:36,740][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:53:37,063][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:53:37,386][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:53:37,710][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:53:38,035][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:53:38,359][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:53:38,682][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:53:39,005][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:53:39,330][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:53:39,655][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:53:39,979][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:53:40,304][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:53:40,629][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:53:40,954][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:53:41,277][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:53:41,602][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:53:41,927][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:53:42,252][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:53:42,576][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:53:42,902][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:53:43,227][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:53:43,912][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:53:44,634][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:53:44,636][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:53:44,637][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:53:45,693][__main__][INFO] - Iteration 586 took 23s (40.14% Gen, 55.35% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 47m 8s. Estimated total time: 19h 34m 5s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 8s, 500 more iterations: 3h 15m 40s.
[2025-11-13 11:53:45,695][__main__][INFO] - Starting iteration 586.
[2025-11-13 11:53:45,699][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:53:45,699][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:53:55,858][__main__][INFO] - Number of regex retries in iteration 586: 0
[2025-11-13 11:53:55,858][__main__][INFO] - agents played in iteration 586 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:53:56,286][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:56,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:56,355][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:56,389][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:53:56,389][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:53:56,390][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:53:57,104][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:53:57,400][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:53:57,727][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:53:58,051][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:53:58,376][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:53:58,701][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:53:59,025][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:53:59,350][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:53:59,679][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:54:00,006][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:54:00,329][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:54:00,659][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:54:00,984][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:54:01,310][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:54:01,637][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:54:01,962][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:54:02,286][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:54:02,612][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:54:02,939][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:54:03,262][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:54:03,588][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:54:03,915][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:54:04,242][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:54:04,565][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:54:04,889][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:54:05,215][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:54:05,537][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:54:05,863][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:54:06,188][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:54:06,515][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:54:06,838][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:54:07,162][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:54:07,489][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:54:08,201][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:54:08,907][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:54:08,908][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:54:08,910][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:54:09,700][__main__][INFO] - Iteration 587 took 24s (42.32% Gen, 54.38% Train). Generation: 10s, Training: 13s. Estimated remaining time: 19h 12m 47s. Estimated total time: 20h 0m 7s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 0s, 500 more iterations: 3h 20m 1s.
[2025-11-13 11:54:09,702][__main__][INFO] - Starting iteration 587.
[2025-11-13 11:54:09,705][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:54:09,705][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:54:18,997][__main__][INFO] - Number of regex retries in iteration 587: 0
[2025-11-13 11:54:18,998][__main__][INFO] - agents played in iteration 587 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:54:19,440][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:19,474][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:19,506][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:19,539][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:19,540][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:54:19,540][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:54:20,236][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:54:20,531][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:54:20,855][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:54:21,184][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:54:21,509][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:54:21,840][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:54:22,170][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:54:22,495][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:54:22,821][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:54:23,147][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:54:23,471][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:54:23,799][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:54:24,125][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:54:24,448][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:54:24,773][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:54:25,096][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:54:25,421][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:54:25,746][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:54:26,070][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:54:26,394][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:54:26,719][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:54:27,043][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:54:27,367][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:54:27,691][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:54:28,014][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:54:28,336][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:54:28,659][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:54:28,981][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:54:29,306][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:54:29,632][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:54:29,957][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:54:30,281][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:54:30,604][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:54:31,306][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:54:32,000][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:54:32,002][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:54:32,004][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:54:32,806][__main__][INFO] - Iteration 588 took 23s (40.22% Gen, 56.30% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 27m 24s. Estimated total time: 19h 15m 8s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 30s, 500 more iterations: 3h 12m 31s.
[2025-11-13 11:54:32,808][__main__][INFO] - Starting iteration 588.
[2025-11-13 11:54:32,811][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:54:32,812][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:54:42,810][__main__][INFO] - Number of regex retries in iteration 588: 0
[2025-11-13 11:54:42,811][__main__][INFO] - agents played in iteration 588 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:54:43,242][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:43,275][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:43,307][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:43,340][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:54:43,341][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:54:43,341][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:54:44,046][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:54:44,339][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:54:44,666][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:54:44,992][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:54:45,322][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:54:45,648][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:54:45,977][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:54:46,304][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:54:46,628][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:54:46,952][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:54:47,278][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:54:47,606][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:54:47,930][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:54:48,255][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:54:48,579][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:54:48,903][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:54:49,228][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:54:49,551][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:54:49,874][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:54:50,198][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:54:50,523][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:54:50,848][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:54:51,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:54:51,500][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:54:51,825][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:54:52,151][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:54:52,477][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:54:52,802][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:54:53,128][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:54:53,455][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:54:53,781][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:54:54,112][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:54:54,435][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:54:55,139][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:54:55,828][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:54:55,830][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:54:55,831][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:54:56,648][__main__][INFO] - Iteration 589 took 23s (41.94% Gen, 54.62% Train). Generation: 9s, Training: 13s. Estimated remaining time: 19h 3m 44s. Estimated total time: 19h 51m 52s. Time estimates for 10 more iterations: 3m 58s, 100 more iterations: 39m 43s, 500 more iterations: 3h 18m 38s.
[2025-11-13 11:54:56,650][__main__][INFO] - Starting iteration 589.
[2025-11-13 11:54:56,653][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:54:56,653][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:55:06,272][__main__][INFO] - Number of regex retries in iteration 589: 0
[2025-11-13 11:55:06,273][__main__][INFO] - agents played in iteration 589 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:55:06,726][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:06,760][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:06,793][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:06,826][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:06,827][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:55:06,827][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:55:07,534][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:55:07,828][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:55:08,152][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:55:08,477][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:55:08,801][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:55:09,130][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:55:09,455][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:55:09,780][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:55:10,107][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:55:10,438][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:55:10,767][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:55:11,092][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:55:11,417][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:55:11,743][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:55:12,068][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:55:12,392][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:55:12,714][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:55:13,037][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:55:13,360][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:55:13,684][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:55:14,009][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:55:14,334][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:55:14,662][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:55:14,989][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:55:15,317][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:55:15,641][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:55:15,967][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:55:16,293][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:55:16,619][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:55:16,946][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:55:17,271][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:55:17,595][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:55:17,922][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:55:18,623][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:55:19,314][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:55:19,315][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:55:19,317][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:55:20,126][__main__][INFO] - Iteration 590 took 23s (40.98% Gen, 55.57% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 45m 9s. Estimated total time: 19h 33m 40s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 7s, 500 more iterations: 3h 15m 36s.
[2025-11-13 11:55:20,127][__main__][INFO] - Starting iteration 590.
[2025-11-13 11:55:20,130][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 58 and human policies 1.
[2025-11-13 11:55:20,131][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:55:29,302][__main__][INFO] - Number of regex retries in iteration 590: 0
[2025-11-13 11:55:29,303][__main__][INFO] - agents played in iteration 590 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:55:29,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:29,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:29,809][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:29,842][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:29,843][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:55:29,843][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:55:30,559][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:55:30,856][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:55:31,183][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:55:31,507][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:55:31,832][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:55:32,155][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:55:32,480][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:55:32,804][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:55:33,128][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:55:33,453][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:55:33,778][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:55:34,102][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:55:34,428][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:55:34,756][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:55:35,083][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:55:35,413][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:55:35,737][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:55:36,064][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:55:36,393][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:55:36,717][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:55:37,043][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:55:37,370][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:55:37,695][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:55:38,020][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:55:38,343][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:55:38,668][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:55:38,995][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:55:39,319][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:55:39,644][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:55:39,968][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:55:40,293][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:55:40,620][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:55:40,944][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:55:41,645][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:55:42,330][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:55:42,332][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:55:42,334][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:55:43,903][__main__][INFO] - Iteration 591 took 23s (38.58% Gen, 54.81% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 59m 47s. Estimated total time: 19h 48m 42s. Time estimates for 10 more iterations: 3m 57s, 100 more iterations: 39m 37s, 500 more iterations: 3h 18m 7s.
[2025-11-13 11:55:43,905][__main__][INFO] - Starting iteration 591.
[2025-11-13 11:55:43,908][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:55:43,909][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:55:53,396][__main__][INFO] - Number of regex retries in iteration 591: 0
[2025-11-13 11:55:53,396][__main__][INFO] - agents played in iteration 591 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:55:53,834][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:53,867][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:53,900][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:53,933][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:55:53,934][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:55:53,934][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:55:54,637][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:55:54,933][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:55:55,258][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:55:55,584][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:55:55,911][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:55:56,235][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:55:56,560][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:55:56,886][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:55:57,212][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:55:57,535][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:55:57,861][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:55:58,188][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:55:58,513][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:55:58,838][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:55:59,163][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:55:59,487][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:55:59,811][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:56:00,135][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:56:00,460][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:56:00,784][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:56:01,108][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:56:01,431][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:56:01,755][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:56:02,078][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:56:02,403][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:56:02,728][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:56:03,052][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:56:03,376][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:56:03,700][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:56:04,026][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:56:04,352][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:56:04,681][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:56:05,006][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:56:05,714][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:56:06,412][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:56:06,413][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:56:06,414][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:56:07,236][__main__][INFO] - Iteration 592 took 23s (40.67% Gen, 55.80% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 37m 8s. Estimated total time: 19h 26m 26s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 52s, 500 more iterations: 3h 14m 24s.
[2025-11-13 11:56:07,238][__main__][INFO] - Starting iteration 592.
[2025-11-13 11:56:07,241][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:56:07,241][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:56:16,553][__main__][INFO] - Number of regex retries in iteration 592: 0
[2025-11-13 11:56:16,553][__main__][INFO] - agents played in iteration 592 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:56:16,997][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:17,029][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:17,062][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:17,095][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:17,096][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:56:17,096][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:56:17,814][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:56:18,109][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:56:18,436][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:56:18,760][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:56:19,086][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:56:19,410][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:56:19,735][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:56:20,058][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:56:20,384][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:56:20,708][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:56:21,034][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:56:21,360][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:56:21,687][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:56:22,014][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:56:22,340][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:56:22,664][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:56:22,988][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:56:23,313][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:56:23,636][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:56:23,961][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:56:24,284][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:56:24,610][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:56:24,935][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:56:25,258][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:56:25,583][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:56:25,908][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:56:26,234][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:56:26,558][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:56:26,884][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:56:27,210][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:56:27,536][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:56:27,860][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:56:28,187][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:56:28,885][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:56:29,583][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:56:29,585][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:56:29,586][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:56:30,419][__main__][INFO] - Iteration 593 took 23s (40.17% Gen, 56.23% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 29m 14s. Estimated total time: 19h 18m 56s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 37s, 500 more iterations: 3h 13m 9s.
[2025-11-13 11:56:30,421][__main__][INFO] - Starting iteration 593.
[2025-11-13 11:56:30,424][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:56:30,424][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:56:39,259][__main__][INFO] - Number of regex retries in iteration 593: 0
[2025-11-13 11:56:39,260][__main__][INFO] - agents played in iteration 593 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:56:39,699][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:39,732][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:39,765][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:39,798][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:56:39,799][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:56:39,799][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:56:40,518][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:56:40,814][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:56:41,139][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:56:41,463][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:56:41,789][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:56:42,115][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:56:42,441][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:56:42,763][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:56:43,089][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:56:43,412][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:56:43,737][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:56:44,060][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:56:44,385][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:56:44,710][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:56:45,032][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:56:45,358][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:56:45,684][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:56:46,011][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:56:46,336][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:56:46,661][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:56:46,987][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:56:47,313][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:56:47,636][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:56:47,959][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:56:48,284][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:56:48,607][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:56:48,932][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:56:49,259][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:56:49,583][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:56:49,908][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:56:50,232][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:56:50,557][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:56:50,881][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:56:51,575][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:56:52,272][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:56:52,273][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:56:52,275][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:56:53,133][__main__][INFO] - Iteration 594 took 22s (38.91% Gen, 57.31% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 5m 25s. Estimated total time: 18h 55m 30s. Time estimates for 10 more iterations: 3m 47s, 100 more iterations: 37m 51s, 500 more iterations: 3h 9m 15s.
[2025-11-13 11:56:53,135][__main__][INFO] - Starting iteration 594.
[2025-11-13 11:56:53,138][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:56:53,138][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:57:02,354][__main__][INFO] - Number of regex retries in iteration 594: 0
[2025-11-13 11:57:02,355][__main__][INFO] - agents played in iteration 594 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:57:02,801][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:02,835][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:02,868][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:02,902][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:02,902][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:57:02,903][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:57:03,617][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:57:03,912][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:57:04,236][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:57:04,559][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:57:04,885][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:57:05,209][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:57:05,533][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:57:05,858][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:57:06,181][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:57:06,506][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:57:06,832][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:57:07,154][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:57:07,479][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:57:07,803][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:57:08,128][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:57:08,452][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:57:08,776][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:57:09,102][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:57:09,428][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:57:09,753][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:57:10,078][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:57:10,404][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:57:10,732][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:57:11,059][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:57:11,385][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:57:11,715][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:57:12,039][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:57:12,365][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:57:12,689][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:57:13,016][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:57:13,341][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:57:13,665][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:57:13,992][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:57:14,708][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:57:15,402][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:57:15,403][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:57:15,405][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:57:16,250][__main__][INFO] - Iteration 595 took 23s (39.87% Gen, 56.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 25m 11s. Estimated total time: 19h 15m 38s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 31s, 500 more iterations: 3h 12m 36s.
[2025-11-13 11:57:16,252][__main__][INFO] - Starting iteration 595.
[2025-11-13 11:57:16,254][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:57:16,255][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:57:25,243][__main__][INFO] - Number of regex retries in iteration 595: 0
[2025-11-13 11:57:25,243][__main__][INFO] - agents played in iteration 595 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:57:25,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:25,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:25,752][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:25,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:25,786][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:57:25,787][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:57:26,508][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:57:26,803][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:57:27,129][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:57:27,452][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:57:27,778][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:57:28,101][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:57:28,425][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:57:28,750][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:57:29,075][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:57:29,403][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:57:29,730][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:57:30,053][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:57:30,377][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:57:30,705][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:57:31,029][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:57:31,353][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:57:31,677][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:57:32,002][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:57:32,327][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:57:32,654][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:57:32,982][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:57:33,310][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:57:33,635][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:57:33,966][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:57:34,295][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:57:34,622][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:57:34,948][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:57:35,273][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:57:35,599][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:57:35,924][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:57:36,249][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:57:36,574][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:57:36,897][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:57:37,593][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:57:38,283][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:57:38,284][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:57:38,286][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:57:39,122][__main__][INFO] - Iteration 596 took 22s (39.31% Gen, 57.03% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 12m 35s. Estimated total time: 19h 3m 25s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 6s, 500 more iterations: 3h 10m 34s.
[2025-11-13 11:57:39,124][__main__][INFO] - Starting iteration 596.
[2025-11-13 11:57:39,127][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:57:39,127][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:57:48,198][__main__][INFO] - Number of regex retries in iteration 596: 0
[2025-11-13 11:57:48,198][__main__][INFO] - agents played in iteration 596 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:57:48,635][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:48,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:48,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:48,735][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:57:48,736][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:57:48,736][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:57:49,450][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:57:49,744][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:57:50,069][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:57:50,394][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:57:50,718][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:57:51,043][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:57:51,369][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:57:51,694][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:57:52,017][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:57:52,343][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:57:52,672][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:57:52,994][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:57:53,319][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:57:53,643][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:57:53,967][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:57:54,291][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:57:54,617][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:57:54,948][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:57:55,274][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:57:55,601][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:57:55,930][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:57:56,255][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:57:56,586][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:57:56,912][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:57:57,240][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:57:57,566][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:57:57,891][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:57:58,215][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:57:58,538][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:57:58,863][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:57:59,187][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:57:59,513][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:57:59,838][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:58:00,558][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:58:01,247][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:58:01,248][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:58:01,251][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:58:02,060][__main__][INFO] - Iteration 597 took 22s (39.55% Gen, 56.91% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 15m 29s. Estimated total time: 19h 6m 42s. Time estimates for 10 more iterations: 3m 49s, 100 more iterations: 38m 13s, 500 more iterations: 3h 11m 7s.
[2025-11-13 11:58:02,062][__main__][INFO] - Starting iteration 597.
[2025-11-13 11:58:02,065][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:58:02,066][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:58:10,786][__main__][INFO] - Number of regex retries in iteration 597: 0
[2025-11-13 11:58:10,786][__main__][INFO] - agents played in iteration 597 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:58:11,222][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:11,255][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:11,288][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:11,322][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:11,322][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:58:11,323][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:58:12,043][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:58:12,340][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:58:12,667][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:58:12,990][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:58:13,315][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:58:13,639][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:58:13,964][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:58:14,287][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:58:14,612][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:58:14,935][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:58:15,260][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:58:15,587][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:58:15,912][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:58:16,234][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:58:16,559][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:58:16,885][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:58:17,209][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:58:17,533][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:58:17,862][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:58:18,190][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:58:18,517][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:58:18,848][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:58:19,175][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:58:19,502][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:58:19,832][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:58:20,158][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:58:20,484][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:58:20,810][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:58:21,134][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:58:21,458][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:58:21,784][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:58:22,108][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:58:22,433][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:58:23,131][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:58:23,829][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:58:23,831][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:58:23,832][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:58:24,669][__main__][INFO] - Iteration 598 took 22s (38.57% Gen, 57.71% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 58m 40s. Estimated total time: 18h 50m 16s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 40s, 500 more iterations: 3h 8m 22s.
[2025-11-13 11:58:24,671][__main__][INFO] - Starting iteration 598.
[2025-11-13 11:58:24,674][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:58:24,675][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:58:33,949][__main__][INFO] - Number of regex retries in iteration 598: 0
[2025-11-13 11:58:33,950][__main__][INFO] - agents played in iteration 598 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:58:34,391][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:34,424][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:34,458][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:34,491][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:34,492][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:58:34,493][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:58:35,213][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:58:35,507][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:58:35,833][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:58:36,158][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:58:36,481][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:58:36,805][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:58:37,130][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:58:37,455][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:58:37,779][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:58:38,104][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:58:38,430][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:58:38,759][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:58:39,086][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:58:39,411][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:58:39,736][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:58:40,062][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:58:40,391][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:58:40,718][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:58:41,042][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:58:41,369][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:58:41,700][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:58:42,024][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:58:42,350][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:58:42,677][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:58:43,004][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:58:43,331][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:58:43,657][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:58:43,981][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:58:44,306][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:58:44,632][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:58:44,955][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:58:45,279][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:58:45,604][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:58:46,308][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:58:46,995][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:58:46,997][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:58:46,998][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:58:48,072][__main__][INFO] - Iteration 599 took 23s (39.64% Gen, 55.77% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 37m 56s. Estimated total time: 19h 29m 55s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 59s, 500 more iterations: 3h 14m 59s.
[2025-11-13 11:58:48,074][__main__][INFO] - Starting iteration 599.
[2025-11-13 11:58:48,077][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:58:48,077][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:58:57,268][__main__][INFO] - Number of regex retries in iteration 599: 0
[2025-11-13 11:58:57,269][__main__][INFO] - agents played in iteration 599 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:58:57,708][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:57,742][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:57,775][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:57,808][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:58:57,808][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:58:57,809][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:58:58,534][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:58:58,831][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:58:59,156][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:58:59,481][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:58:59,805][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:59:00,130][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:59:00,454][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:59:00,779][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:59:01,106][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:59:01,428][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:59:01,753][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:59:02,077][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:59:02,404][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:59:02,728][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:59:03,051][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:59:03,375][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:59:03,699][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:59:04,029][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:59:04,358][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:59:04,684][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:59:05,009][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:59:05,336][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:59:05,662][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:59:05,994][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:59:06,324][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:59:06,649][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:59:06,975][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:59:07,300][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:59:07,625][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:59:07,949][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:59:08,273][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:59:08,596][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:59:08,920][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:59:09,627][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:59:10,318][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:59:10,319][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:59:10,321][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:59:11,477][__main__][INFO] - Iteration 600 took 23s (39.28% Gen, 55.77% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 37m 42s. Estimated total time: 19h 30m 4s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 0s, 500 more iterations: 3h 15m 0s.
[2025-11-13 11:59:11,479][__main__][INFO] - Starting iteration 600.
[2025-11-13 11:59:11,483][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 59 and human policies 1.
[2025-11-13 11:59:11,484][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:59:20,186][__main__][INFO] - Number of regex retries in iteration 600: 0
[2025-11-13 11:59:20,186][__main__][INFO] - agents played in iteration 600 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:59:20,634][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:20,668][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:20,701][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:20,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:20,735][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:59:20,735][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:59:21,461][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:59:21,755][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:59:22,079][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:59:22,405][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:59:22,728][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:59:23,053][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:59:23,380][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:59:23,703][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:59:24,028][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:59:24,353][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:59:24,679][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:59:25,004][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:59:25,332][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:59:25,660][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:59:25,987][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:59:26,318][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:59:26,644][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:59:26,970][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:59:27,294][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:59:27,617][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:59:27,942][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:59:28,269][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:59:28,594][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:59:28,918][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:59:29,243][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:59:29,568][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:59:29,898][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:59:30,222][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:59:30,548][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:59:30,873][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:59:31,199][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:59:31,525][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:59:31,849][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:59:32,542][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:59:33,239][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:59:33,241][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:59:33,243][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:59:35,397][__main__][INFO] - Iteration 601 took 23s (36.39% Gen, 54.60% Train). Generation: 8s, Training: 13s. Estimated remaining time: 19h 2m 57s. Estimated total time: 19h 55m 43s. Time estimates for 10 more iterations: 3m 59s, 100 more iterations: 39m 51s, 500 more iterations: 3h 19m 17s.
[2025-11-13 11:59:35,399][__main__][INFO] - Starting iteration 601.
[2025-11-13 11:59:35,403][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 11:59:35,403][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 11:59:44,234][__main__][INFO] - Number of regex retries in iteration 601: 0
[2025-11-13 11:59:44,235][__main__][INFO] - agents played in iteration 601 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 11:59:44,676][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:44,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:44,743][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:44,776][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 11:59:44,777][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 11:59:44,778][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 11:59:45,511][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 11:59:45,806][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 11:59:46,132][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 11:59:46,457][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 11:59:46,784][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 11:59:47,109][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 11:59:47,433][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 11:59:47,757][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 11:59:48,082][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 11:59:48,407][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 11:59:48,731][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 11:59:49,054][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 11:59:49,379][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 11:59:49,703][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 11:59:50,027][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 11:59:50,355][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 11:59:50,680][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 11:59:51,007][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 11:59:51,332][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 11:59:51,656][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 11:59:51,982][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 11:59:52,308][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 11:59:52,632][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 11:59:52,957][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 11:59:53,284][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 11:59:53,610][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 11:59:53,936][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 11:59:54,260][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 11:59:54,585][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 11:59:54,910][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 11:59:55,234][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 11:59:55,559][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 11:59:55,885][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 11:59:56,598][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 11:59:57,288][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 11:59:57,290][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 11:59:57,291][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 11:59:58,441][__main__][INFO] - Iteration 602 took 23s (38.33% Gen, 56.67% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 18m 48s. Estimated total time: 19h 11m 58s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 23s, 500 more iterations: 3h 11m 59s.
[2025-11-13 11:59:58,443][__main__][INFO] - Starting iteration 602.
[2025-11-13 11:59:58,446][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 11:59:58,447][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:00:07,171][__main__][INFO] - Number of regex retries in iteration 602: 0
[2025-11-13 12:00:07,172][__main__][INFO] - agents played in iteration 602 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:00:07,610][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:07,644][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:07,677][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:07,710][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:07,711][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:00:07,711][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:00:08,436][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:00:08,733][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:00:09,059][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:00:09,384][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:00:09,708][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:00:10,031][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:00:10,355][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:00:10,679][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:00:11,003][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:00:11,328][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:00:11,654][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:00:11,977][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:00:12,302][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:00:12,627][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:00:12,955][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:00:13,281][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:00:13,609][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:00:13,937][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:00:14,261][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:00:14,591][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:00:14,917][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:00:15,246][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:00:15,573][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:00:15,896][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:00:16,223][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:00:16,551][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:00:16,879][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:00:17,205][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:00:17,529][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:00:17,857][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:00:18,182][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:00:18,508][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:00:18,833][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:00:19,536][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:00:20,221][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:00:20,223][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:00:20,224][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:00:21,078][__main__][INFO] - Iteration 603 took 22s (38.55% Gen, 57.67% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 58m 8s. Estimated total time: 18h 51m 40s. Time estimates for 10 more iterations: 3m 46s, 100 more iterations: 37m 43s, 500 more iterations: 3h 8m 36s.
[2025-11-13 12:00:21,081][__main__][INFO] - Starting iteration 603.
[2025-11-13 12:00:21,084][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:00:21,085][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:00:30,243][__main__][INFO] - Number of regex retries in iteration 603: 0
[2025-11-13 12:00:30,243][__main__][INFO] - agents played in iteration 603 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:00:30,684][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:30,718][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:30,751][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:30,785][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:30,786][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:00:30,787][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:00:31,511][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:00:31,805][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:00:32,131][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:00:32,454][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:00:32,778][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:00:33,102][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:00:33,426][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:00:33,751][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:00:34,074][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:00:34,399][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:00:34,723][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:00:35,047][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:00:35,372][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:00:35,698][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:00:36,024][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:00:36,347][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:00:36,672][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:00:36,997][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:00:37,324][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:00:37,648][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:00:37,972][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:00:38,297][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:00:38,622][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:00:38,948][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:00:39,276][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:00:39,602][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:00:39,927][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:00:40,253][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:00:40,577][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:00:40,902][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:00:41,227][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:00:41,552][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:00:41,877][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:00:42,576][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:00:43,317][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:00:43,319][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:00:43,321][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:00:44,435][__main__][INFO] - Iteration 604 took 23s (39.22% Gen, 56.00% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 33m 40s. Estimated total time: 19h 27m 35s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 55s, 500 more iterations: 3h 14m 35s.
[2025-11-13 12:00:44,437][__main__][INFO] - Starting iteration 604.
[2025-11-13 12:00:44,440][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:00:44,440][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:00:53,530][__main__][INFO] - Number of regex retries in iteration 604: 0
[2025-11-13 12:00:53,530][__main__][INFO] - agents played in iteration 604 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:00:53,968][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:54,003][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:54,037][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:54,070][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:00:54,071][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:00:54,071][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:00:54,775][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:00:55,070][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:00:55,394][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:00:55,719][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:00:56,043][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:00:56,367][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:00:56,692][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:00:57,014][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:00:57,339][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:00:57,666][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:00:57,990][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:00:58,314][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:00:58,639][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:00:58,962][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:00:59,287][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:00:59,612][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:00:59,938][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:01:00,265][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:01:00,589][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:01:00,917][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:01:01,242][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:01:01,569][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:01:01,893][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:01:02,219][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:01:02,550][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:01:02,876][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:01:03,202][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:01:03,526][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:01:03,852][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:01:04,176][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:01:04,502][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:01:04,827][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:01:05,151][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:01:05,878][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:01:06,571][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:01:06,573][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:01:06,575][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:01:07,961][__main__][INFO] - Iteration 605 took 23s (38.64% Gen, 55.46% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 41m 47s. Estimated total time: 19h 36m 5s. Time estimates for 10 more iterations: 3m 55s, 100 more iterations: 39m 12s, 500 more iterations: 3h 16m 0s.
[2025-11-13 12:01:07,963][__main__][INFO] - Starting iteration 605.
[2025-11-13 12:01:07,966][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:01:07,966][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:01:16,763][__main__][INFO] - Number of regex retries in iteration 605: 0
[2025-11-13 12:01:16,763][__main__][INFO] - agents played in iteration 605 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:01:17,204][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:17,237][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:17,271][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:17,306][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:17,306][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:01:17,307][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:01:18,011][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:01:18,306][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:01:18,634][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:01:18,962][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:01:19,288][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:01:19,613][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:01:19,937][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:01:20,262][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:01:20,587][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:01:20,912][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:01:21,237][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:01:21,562][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:01:21,887][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:01:22,213][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:01:22,536][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:01:22,861][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:01:23,186][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:01:23,512][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:01:23,838][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:01:24,162][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:01:24,487][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:01:24,811][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:01:25,136][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:01:25,460][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:01:25,788][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:01:26,115][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:01:26,440][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:01:26,764][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:01:27,088][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:01:27,413][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:01:27,737][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:01:28,063][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:01:28,392][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:01:29,218][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:01:29,931][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:01:29,932][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:01:29,934][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:01:30,840][__main__][INFO] - Iteration 606 took 22s (38.46% Gen, 57.58% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 9m 4s. Estimated total time: 19h 3m 45s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 7s, 500 more iterations: 3h 10m 37s.
[2025-11-13 12:01:30,842][__main__][INFO] - Starting iteration 606.
[2025-11-13 12:01:30,846][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:01:30,846][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:01:39,426][__main__][INFO] - Number of regex retries in iteration 606: 0
[2025-11-13 12:01:39,426][__main__][INFO] - agents played in iteration 606 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:01:39,865][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:39,899][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:39,932][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:39,965][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:01:39,965][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:01:39,966][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:01:40,687][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:01:40,982][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:01:41,310][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:01:41,634][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:01:41,957][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:01:42,282][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:01:42,607][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:01:42,932][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:01:43,256][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:01:43,581][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:01:43,905][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:01:44,229][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:01:44,554][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:01:44,877][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:01:45,202][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:01:45,526][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:01:45,851][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:01:46,176][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:01:46,501][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:01:46,826][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:01:47,149][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:01:47,474][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:01:47,801][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:01:48,126][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:01:48,451][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:01:48,780][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:01:49,106][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:01:49,431][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:01:49,756][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:01:50,082][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:01:50,407][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:01:50,730][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:01:51,054][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:01:51,780][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:01:52,482][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:01:52,483][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:01:52,485][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:01:53,862][__main__][INFO] - Iteration 607 took 23s (37.28% Gen, 56.73% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 15m 47s. Estimated total time: 19h 10m 52s. Time estimates for 10 more iterations: 3m 50s, 100 more iterations: 38m 21s, 500 more iterations: 3h 11m 48s.
[2025-11-13 12:01:53,864][__main__][INFO] - Starting iteration 607.
[2025-11-13 12:01:53,867][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:01:53,868][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:02:02,694][__main__][INFO] - Number of regex retries in iteration 607: 0
[2025-11-13 12:02:02,695][__main__][INFO] - agents played in iteration 607 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:02:03,125][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:03,162][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:03,195][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:03,228][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:03,229][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:02:03,229][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:02:03,948][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:02:04,244][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:02:04,569][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:02:04,896][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:02:05,226][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:02:05,552][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:02:05,877][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:02:06,202][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:02:06,527][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:02:06,852][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:02:07,177][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:02:07,503][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:02:07,828][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:02:08,153][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:02:08,477][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:02:08,803][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:02:09,128][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:02:09,454][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:02:09,779][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:02:10,105][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:02:10,430][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:02:10,756][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:02:11,081][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:02:11,405][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:02:11,731][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:02:12,055][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:02:12,379][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:02:12,705][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:02:13,030][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:02:13,353][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:02:13,679][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:02:14,003][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:02:14,327][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:02:15,051][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:02:15,749][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:02:15,754][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:02:15,755][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:02:16,726][__main__][INFO] - Iteration 608 took 22s (38.61% Gen, 57.13% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 7m 31s. Estimated total time: 19h 2m 59s. Time estimates for 10 more iterations: 3m 48s, 100 more iterations: 38m 5s, 500 more iterations: 3h 10m 29s.
[2025-11-13 12:02:16,728][__main__][INFO] - Starting iteration 608.
[2025-11-13 12:02:16,731][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:02:16,732][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:02:25,486][__main__][INFO] - Number of regex retries in iteration 608: 0
[2025-11-13 12:02:25,486][__main__][INFO] - agents played in iteration 608 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:02:25,923][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:25,959][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:25,992][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:26,025][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:26,026][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:02:26,026][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:02:26,716][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:02:27,012][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:02:27,338][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:02:27,662][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:02:27,988][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:02:28,312][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:02:28,640][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:02:28,964][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:02:29,292][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:02:29,617][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:02:29,942][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:02:30,267][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:02:30,592][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:02:30,916][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:02:31,240][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:02:31,565][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:02:31,889][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:02:32,216][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:02:32,540][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:02:32,866][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:02:33,190][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:02:33,516][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:02:33,841][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:02:34,166][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:02:34,490][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:02:34,814][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:02:35,139][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:02:35,465][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:02:35,790][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:02:36,113][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:02:36,438][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:02:36,764][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:02:37,090][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:02:37,811][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:02:38,508][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:02:38,510][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:02:38,511][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:02:39,848][__main__][INFO] - Iteration 609 took 23s (37.87% Gen, 56.34% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 20m 1s. Estimated total time: 19h 15m 52s. Time estimates for 10 more iterations: 3m 51s, 100 more iterations: 38m 31s, 500 more iterations: 3h 12m 38s.
[2025-11-13 12:02:39,850][__main__][INFO] - Starting iteration 609.
[2025-11-13 12:02:39,853][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:02:39,854][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:02:48,243][__main__][INFO] - Number of regex retries in iteration 609: 0
[2025-11-13 12:02:48,244][__main__][INFO] - agents played in iteration 609 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:02:48,669][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:48,702][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:48,734][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:48,767][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:02:48,768][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:02:48,768][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:02:49,442][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:02:49,737][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:02:50,063][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:02:50,386][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:02:50,710][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:02:51,035][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:02:51,359][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:02:51,686][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:02:52,012][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:02:52,338][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:02:52,662][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:02:52,989][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:02:53,313][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:02:53,639][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:02:53,963][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:02:54,286][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:02:54,611][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:02:54,934][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:02:55,260][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:02:55,585][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:02:55,910][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:02:56,237][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:02:56,562][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:02:56,886][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:02:57,212][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:02:57,537][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:02:57,862][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:02:58,187][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:02:58,513][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:02:58,838][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:02:59,164][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:02:59,486][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:02:59,810][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:03:00,517][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:03:01,213][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:03:01,215][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:03:01,216][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:03:02,295][__main__][INFO] - Iteration 610 took 22s (37.38% Gen, 57.81% Train). Generation: 8s, Training: 12s. Estimated remaining time: 17h 45m 53s. Estimated total time: 18h 42m 6s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 24s, 500 more iterations: 3h 7m 1s.
[2025-11-13 12:03:02,297][__main__][INFO] - Starting iteration 610.
[2025-11-13 12:03:02,300][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 60 and human policies 1.
[2025-11-13 12:03:02,300][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:03:11,235][__main__][INFO] - Number of regex retries in iteration 610: 0
[2025-11-13 12:03:11,235][__main__][INFO] - agents played in iteration 610 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:03:11,655][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:11,688][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:11,720][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:11,753][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:11,753][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:03:11,754][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:03:12,428][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:03:12,722][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:03:13,051][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:03:13,375][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:03:13,703][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:03:14,030][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:03:14,357][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:03:14,685][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:03:15,010][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:03:15,335][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:03:15,659][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:03:15,984][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:03:16,308][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:03:16,631][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:03:16,955][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:03:17,282][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:03:17,608][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:03:17,931][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:03:18,256][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:03:18,580][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:03:18,906][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:03:19,230][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:03:19,553][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:03:19,878][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:03:20,203][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:03:20,530][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:03:20,855][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:03:21,179][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:03:21,505][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:03:21,831][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:03:22,155][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:03:22,485][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:03:22,809][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:03:23,532][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:03:24,229][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:03:24,231][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:03:24,233][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:03:26,307][__main__][INFO] - Iteration 611 took 24s (37.22% Gen, 54.14% Train). Generation: 8s, Training: 12s. Estimated remaining time: 19h 3m 48s. Estimated total time: 20h 0m 25s. Time estimates for 10 more iterations: 4m 0s, 100 more iterations: 40m 0s, 500 more iterations: 3h 20m 4s.
[2025-11-13 12:03:26,309][__main__][INFO] - Starting iteration 611.
[2025-11-13 12:03:26,312][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 12:03:26,313][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:03:34,894][__main__][INFO] - Number of regex retries in iteration 611: 0
[2025-11-13 12:03:34,894][__main__][INFO] - agents played in iteration 611 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:03:35,318][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:35,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:35,386][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:35,419][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:35,419][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:03:35,420][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:03:36,097][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:03:36,394][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:03:36,721][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:03:37,048][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:03:37,375][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:03:37,703][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:03:38,028][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:03:38,354][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:03:38,685][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:03:39,010][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:03:39,338][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:03:39,668][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:03:39,993][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:03:40,317][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:03:40,642][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:03:40,965][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:03:41,288][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:03:41,611][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:03:41,936][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:03:42,259][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:03:42,584][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:03:42,909][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:03:43,234][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:03:43,557][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:03:43,882][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:03:44,207][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:03:44,533][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:03:44,857][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:03:45,181][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:03:45,506][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:03:45,832][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:03:46,158][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:03:46,485][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:03:47,184][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:03:47,900][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:03:47,901][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:03:47,903][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:03:48,794][__main__][INFO] - Iteration 612 took 22s (38.16% Gen, 57.86% Train). Generation: 8s, Training: 13s. Estimated remaining time: 17h 47m 9s. Estimated total time: 18h 44m 8s. Time estimates for 10 more iterations: 3m 44s, 100 more iterations: 37m 28s, 500 more iterations: 3h 7m 21s.
[2025-11-13 12:03:48,796][__main__][INFO] - Starting iteration 612.
[2025-11-13 12:03:48,799][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 12:03:48,799][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:03:58,006][__main__][INFO] - Number of regex retries in iteration 612: 0
[2025-11-13 12:03:58,007][__main__][INFO] - agents played in iteration 612 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:03:58,429][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:58,462][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:58,495][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:58,527][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:03:58,528][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:03:58,528][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:03:59,207][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:03:59,501][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:03:59,825][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:04:00,149][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:04:00,472][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:04:00,796][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:04:01,122][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:04:01,446][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:04:01,771][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:04:02,096][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:04:02,420][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:04:02,747][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:04:03,072][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:04:03,397][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:04:03,721][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:04:04,046][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:04:04,371][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:04:04,695][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:04:05,019][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:04:05,344][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:04:05,669][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:04:05,993][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:04:06,317][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:04:06,641][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:04:06,965][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:04:07,291][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:04:07,616][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:04:07,940][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:04:08,264][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:04:08,589][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:04:08,914][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:04:09,238][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:04:09,565][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:04:10,266][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:04:10,982][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:04:10,983][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:04:10,985][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:04:12,132][__main__][INFO] - Iteration 613 took 23s (39.46% Gen, 55.62% Train). Generation: 9s, Training: 12s. Estimated remaining time: 18h 29m 19s. Estimated total time: 19h 26m 42s. Time estimates for 10 more iterations: 3m 53s, 100 more iterations: 38m 53s, 500 more iterations: 3h 14m 27s.
[2025-11-13 12:04:12,134][__main__][INFO] - Starting iteration 613.
[2025-11-13 12:04:12,137][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 12:04:12,138][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:04:21,100][__main__][INFO] - Number of regex retries in iteration 613: 0
[2025-11-13 12:04:21,101][__main__][INFO] - agents played in iteration 613 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:04:21,535][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:21,569][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:21,602][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:21,636][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:21,636][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:04:21,637][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:04:22,330][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:04:22,625][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:04:22,949][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:04:23,272][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:04:23,602][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:04:23,929][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:04:24,254][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:04:24,577][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:04:24,902][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:04:25,228][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:04:25,552][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:04:25,877][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:04:26,203][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:04:26,528][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:04:26,856][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:04:27,181][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:04:27,505][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:04:27,830][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:04:28,154][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:04:28,480][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:04:28,802][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:04:29,126][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:04:29,452][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:04:29,774][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:04:30,098][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:04:30,422][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:04:30,748][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:04:31,072][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:04:31,398][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:04:31,724][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:04:32,049][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:04:32,375][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:04:32,699][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:04:33,409][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:04:34,136][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:04:34,137][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:04:34,138][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:04:35,575][__main__][INFO] - Iteration 614 took 23s (38.24% Gen, 55.62% Train). Generation: 8s, Training: 13s. Estimated remaining time: 18h 34m 9s. Estimated total time: 19h 31m 56s. Time estimates for 10 more iterations: 3m 54s, 100 more iterations: 39m 3s, 500 more iterations: 3h 15m 19s.
[2025-11-13 12:04:35,577][__main__][INFO] - Starting iteration 614.
[2025-11-13 12:04:35,580][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 12:04:35,581][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:04:44,855][__main__][INFO] - Number of regex retries in iteration 614: 0
[2025-11-13 12:04:44,856][__main__][INFO] - agents played in iteration 614 are Alice_buffer, Alice, Bob_buffer, Bob
[2025-11-13 12:04:45,287][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:45,320][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:45,354][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:45,388][mllm.training.trainer_ad_align][INFO] - For task: Get advantages with critic gradient accumulation, ΔVRAM % (total): 0.00%, Current % of VRAM taken: 40.57%, Block Peak % of device VRAM: 19.52%, ΔTime: 00:00:00
[2025-11-13 12:04:45,388][mllm.training.trainer_ad_align][INFO] - Sharing advantage alignment data.
[2025-11-13 12:04:45,389][mllm.training.trainer_ad_align][INFO] - Receiving advantage packets.
[2025-11-13 12:04:46,062][mllm.training.trainer_common][INFO] - Processing mini-batch 0 of 128
[2025-11-13 12:04:46,356][mllm.training.trainer_common][INFO] - Processing mini-batch 4 of 128
[2025-11-13 12:04:46,682][mllm.training.trainer_common][INFO] - Processing mini-batch 8 of 128
[2025-11-13 12:04:47,007][mllm.training.trainer_common][INFO] - Processing mini-batch 12 of 128
[2025-11-13 12:04:47,332][mllm.training.trainer_common][INFO] - Processing mini-batch 16 of 128
[2025-11-13 12:04:47,655][mllm.training.trainer_common][INFO] - Processing mini-batch 20 of 128
[2025-11-13 12:04:47,980][mllm.training.trainer_common][INFO] - Processing mini-batch 24 of 128
[2025-11-13 12:04:48,306][mllm.training.trainer_common][INFO] - Processing mini-batch 28 of 128
[2025-11-13 12:04:48,630][mllm.training.trainer_common][INFO] - Processing mini-batch 32 of 128
[2025-11-13 12:04:48,957][mllm.training.trainer_common][INFO] - Processing mini-batch 36 of 128
[2025-11-13 12:04:49,283][mllm.training.trainer_common][INFO] - Processing mini-batch 40 of 128
[2025-11-13 12:04:49,611][mllm.training.trainer_common][INFO] - Processing mini-batch 44 of 128
[2025-11-13 12:04:49,935][mllm.training.trainer_common][INFO] - Processing mini-batch 48 of 128
[2025-11-13 12:04:50,258][mllm.training.trainer_common][INFO] - Processing mini-batch 52 of 128
[2025-11-13 12:04:50,583][mllm.training.trainer_common][INFO] - Processing mini-batch 56 of 128
[2025-11-13 12:04:50,908][mllm.training.trainer_common][INFO] - Processing mini-batch 60 of 128
[2025-11-13 12:04:51,233][mllm.training.trainer_common][INFO] - Processing mini-batch 64 of 128
[2025-11-13 12:04:51,557][mllm.training.trainer_common][INFO] - Processing mini-batch 68 of 128
[2025-11-13 12:04:51,881][mllm.training.trainer_common][INFO] - Processing mini-batch 72 of 128
[2025-11-13 12:04:52,205][mllm.training.trainer_common][INFO] - Processing mini-batch 76 of 128
[2025-11-13 12:04:52,528][mllm.training.trainer_common][INFO] - Processing mini-batch 80 of 128
[2025-11-13 12:04:52,852][mllm.training.trainer_common][INFO] - Processing mini-batch 84 of 128
[2025-11-13 12:04:53,176][mllm.training.trainer_common][INFO] - Processing mini-batch 88 of 128
[2025-11-13 12:04:53,500][mllm.training.trainer_common][INFO] - Processing mini-batch 92 of 128
[2025-11-13 12:04:53,825][mllm.training.trainer_common][INFO] - Processing mini-batch 96 of 128
[2025-11-13 12:04:54,148][mllm.training.trainer_common][INFO] - Processing mini-batch 100 of 128
[2025-11-13 12:04:54,472][mllm.training.trainer_common][INFO] - Processing mini-batch 104 of 128
[2025-11-13 12:04:54,796][mllm.training.trainer_common][INFO] - Processing mini-batch 108 of 128
[2025-11-13 12:04:55,122][mllm.training.trainer_common][INFO] - Processing mini-batch 112 of 128
[2025-11-13 12:04:55,447][mllm.training.trainer_common][INFO] - Processing mini-batch 116 of 128
[2025-11-13 12:04:55,771][mllm.training.trainer_common][INFO] - Processing mini-batch 120 of 128
[2025-11-13 12:04:56,095][mllm.training.trainer_common][INFO] - Processing mini-batch 124 of 128
[2025-11-13 12:04:56,421][mllm.training.trainer_common][INFO] - Accumulated the policy gradient loss for 3840 tokens.
[2025-11-13 12:04:57,134][mllm.training.trainer_common][INFO] - For task: Apply reinforce step, ΔVRAM % (total): 2.51%, Current % of VRAM taken: 41.82%, Block Peak % of device VRAM: 25.97%, ΔTime: 00:00:11
[2025-11-13 12:04:57,853][mllm.training.trainer_common][INFO] - Saved main optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/policy_optimizer_state.pt
[2025-11-13 12:04:57,854][mllm.training.trainer_common][INFO] - Saved critic optimizer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/critic_optimizer_state.pt
[2025-11-13 12:04:57,856][mllm.training.trainer_common][INFO] - Saved trainer state to /scratch/m/muqeeth/llm_negotiation/2025_11/ipd_ad_align_nocurrtimestep_seed1_bs128/seed_1/agent_trainer/trainer_annealing_state.pkl
[2025-11-13 12:04:58,817][__main__][INFO] - Iteration 615 took 23s (39.91% Gen, 55.95% Train). Generation: 9s, Training: 13s. Estimated remaining time: 18h 23m 42s. Estimated total time: 19h 21m 52s. Time estimates for 10 more iterations: 3m 52s, 100 more iterations: 38m 43s, 500 more iterations: 3h 13m 38s.
[2025-11-13 12:04:58,819][__main__][INFO] - Starting iteration 615.
[2025-11-13 12:04:58,821][__main__][INFO] - Inference policies count is regular policies 2 and buffer policies 61 and human policies 1.
[2025-11-13 12:04:58,822][__main__][INFO] - Hard coded buffer agents are set to False with prob 0
[2025-11-13 12:05:10,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,619][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,620][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,621][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,622][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,623][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,624][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,625][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,626][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,627][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,628][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,629][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,630][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,631][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,665][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,666][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,667][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,668][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,669][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,670][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,671][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,672][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,673][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,674][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,675][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,676][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,677][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,678][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,679][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,680][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,681][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,682][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,683][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,684][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,685][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,686][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,687][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,688][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,689][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,690][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,691][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,692][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,693][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,694][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,695][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,696][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,697][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,698][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,699][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,700][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,701][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,702][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,703][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,704][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,705][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,706][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,707][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,708][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,709][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,710][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,711][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,712][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,713][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,714][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,715][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,716][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,717][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,718][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,719][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,720][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,721][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,722][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,723][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,724][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,725][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,726][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,727][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,728][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,729][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,730][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,731][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,732][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,733][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,734][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,735][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,736][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,737][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,738][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,739][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,740][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,741][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,742][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,743][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,744][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,745][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,746][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,747][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,748][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,749][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,750][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,751][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,752][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,753][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,754][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,755][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,756][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,757][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,758][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,759][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,760][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,761][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,762][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,763][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,764][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,765][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,766][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,767][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,845][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,846][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,847][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,848][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,849][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,850][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,851][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,852][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,853][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,854][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,855][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,856][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,857][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,858][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,859][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,860][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,861][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,862][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,863][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,864][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,865][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,866][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,867][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,868][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,869][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,870][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,871][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,872][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,873][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,874][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,875][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,876][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,877][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,878][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,879][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,880][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,881][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,882][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,883][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,884][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,885][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,886][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,887][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,888][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,889][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,890][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,891][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,892][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,893][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,894][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,895][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,896][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,897][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,898][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,899][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,900][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,901][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,902][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,903][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,904][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,905][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,906][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,907][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,908][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,909][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,910][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,911][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,912][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,913][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,914][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,915][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,916][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,917][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,918][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,919][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,920][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,921][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,922][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,923][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,924][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,925][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,926][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,927][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,928][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:10,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,208][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,209][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,210][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,211][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,212][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,213][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,214][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,215][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,216][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,217][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,218][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,219][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,220][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,221][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,222][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,223][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,224][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,225][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,226][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,227][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,228][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,229][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,230][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,231][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,232][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,233][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,234][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,235][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,236][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,237][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,238][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,239][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,240][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,241][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,242][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,243][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,244][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,245][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,246][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,247][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,248][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,249][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,250][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,251][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,252][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,253][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,254][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,255][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,256][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,257][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,258][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,259][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,260][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,261][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,262][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,263][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,264][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,265][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,266][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,267][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,268][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,269][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,270][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,271][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,272][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,273][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,274][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,275][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,276][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,277][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,278][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,279][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,280][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,281][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,282][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,283][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,284][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,285][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,286][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,287][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,288][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,289][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,290][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,291][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,292][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,293][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,294][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,295][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,296][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,297][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,298][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,299][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,300][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,301][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,302][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,303][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,304][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,305][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,306][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,307][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,308][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,309][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,310][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,311][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,312][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,313][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,314][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,315][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,316][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,317][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,318][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,319][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,320][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,321][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,322][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,323][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,324][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,325][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,326][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,327][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,328][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,329][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,330][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,331][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,332][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,333][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,334][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,335][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,336][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,337][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,338][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,339][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,340][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,341][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,342][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,343][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,344][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,345][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,346][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,347][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,348][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,349][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,350][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,351][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,352][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,353][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,354][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,355][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,356][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,357][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,358][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,359][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,360][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,361][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,362][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,363][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,364][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,365][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,366][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,367][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,368][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,369][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,370][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,371][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,372][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,373][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,374][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,375][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,376][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,377][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,378][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,379][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,380][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,381][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,382][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,383][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,384][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,385][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,386][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,387][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,388][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,389][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,390][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,391][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,392][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,393][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,394][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,395][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,396][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,397][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,398][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,399][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,400][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,401][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,402][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,403][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,404][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,405][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,406][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,407][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,408][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,409][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,410][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,411][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,412][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,413][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,414][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,415][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,416][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,417][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,418][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,419][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,420][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,421][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,422][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,423][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,424][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,425][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,426][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,427][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,428][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,429][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,430][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,431][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,432][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,433][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,434][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,435][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,436][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,437][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,438][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,439][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,440][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,441][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,442][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,443][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,444][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,445][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,446][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,447][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,448][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,449][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,450][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,451][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,452][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,453][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,454][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,455][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,456][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,457][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,458][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,459][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,460][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,461][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,462][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,463][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,464][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,465][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,466][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,467][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,468][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,469][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,470][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,471][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,472][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,473][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,474][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,475][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,476][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,477][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,478][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,479][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,480][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,481][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,482][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,483][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,484][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,485][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,486][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,487][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,488][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,489][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,490][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,491][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,492][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,493][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,494][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,495][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,496][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,497][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,498][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,499][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,500][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,501][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,502][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,503][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,504][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,505][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,506][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,507][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,508][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,509][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,510][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,511][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,512][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,513][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,514][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,515][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,516][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,517][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,518][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,519][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,520][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,521][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,522][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,523][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,524][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,525][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,526][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,527][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,528][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,529][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,530][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,531][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,532][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,533][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,534][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,535][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,536][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,537][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,538][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,539][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,540][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,541][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,542][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,543][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,544][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,545][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,546][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,547][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,548][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,549][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,550][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,551][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,552][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,553][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,554][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,555][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,556][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,557][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,558][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,559][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,560][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,561][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,562][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,563][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,564][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,565][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,566][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,567][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,568][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,569][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,570][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,571][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,572][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,573][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,574][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,575][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,576][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,577][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,578][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,579][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,580][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,581][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,582][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,583][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,584][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,585][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,586][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,587][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,588][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,589][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,590][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,591][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,592][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,593][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,594][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,595][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,596][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,597][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,598][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,599][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,600][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,601][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,602][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,603][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,604][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,605][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,606][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,607][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,608][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,609][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,610][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,611][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,612][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,613][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,614][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,615][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,616][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,617][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,618][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,929][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,930][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,931][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,932][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,933][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,934][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,935][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,936][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,937][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,938][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,939][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,940][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,941][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,942][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,943][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,944][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,945][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,946][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,947][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,948][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,949][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,950][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,951][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,952][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,953][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,954][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,955][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,956][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,957][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,958][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,959][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,960][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,961][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,962][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,963][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,964][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,965][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,966][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,967][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,968][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,969][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,970][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,971][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,972][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,973][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,974][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,975][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,976][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,977][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,978][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,979][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,980][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,981][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,982][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,983][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,984][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,985][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,986][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,987][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,988][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,989][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,990][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,991][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,992][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,993][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,994][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,995][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,996][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,997][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,998][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:11,999][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,000][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,001][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,002][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,003][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,004][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,005][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,006][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,007][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,008][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,009][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,010][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,011][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,012][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,013][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,014][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,015][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,016][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,017][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,018][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,019][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,020][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,021][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,022][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,023][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,024][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,025][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,026][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,027][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,028][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,029][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,030][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,031][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,032][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,033][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,034][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,035][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,036][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,037][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,038][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,039][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,040][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,041][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,042][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,043][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,044][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,045][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,046][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,047][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,048][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,049][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,050][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,051][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,052][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,053][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,054][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,055][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,056][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,057][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,058][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,059][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,060][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,061][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,062][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,063][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,064][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,065][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,066][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,067][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,068][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,069][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,070][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,071][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,072][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,073][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,074][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,075][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,076][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,077][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,078][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,079][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,080][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,081][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,082][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,083][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,084][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,085][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,086][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,087][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,088][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,089][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,090][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,091][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,092][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,093][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,094][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,095][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,096][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,097][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,098][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,099][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,100][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,101][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,102][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,103][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,104][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,105][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,106][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,107][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,108][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,109][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,110][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,111][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,112][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,113][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,114][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,115][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,116][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,117][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,118][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,119][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,120][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,121][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,122][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,123][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,124][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,125][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,126][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,129][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,130][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,131][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,132][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,133][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,138][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,139][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,140][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,141][asyncio][WARNING] - socket.send() raised exception.
[2025-11-13 12:05:12,241][asyncio][WARNING] - socket.send() raised exception.